What can cause this error on one server, but not another?

Question

We have an ASP.Net website that connects to an external web service, and it abruptly stopped working a few days ago.

The basic code is this:

Try
    request = New ExternalWebService.ProcessRequestService

    ' Error occurs here:
    response = request.processCommand(parameters)

    ' Do some other stuff with response here

Catch webEx As System.Net.WebException

    ' This triggers and generic error gets displayed

Catch ex As Exception

Finally

End Try

The System.Net.WebException gets triggered with a SendFailure exception, and returns this stack trace:

The underlying connection was closed: An unexpected error occurred on a send. -
    at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
    at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)
    at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
    at ExternalWebService.ProcessRequestService.processCommand(parameters)

When I try to debug this problem on my development machine, it works fine and no exception occurs.

There has been no recent updates to our webserver, and I can connect to the wsdl service from the web server with FireFox (I cannot connect with IE however, but that might be related to another issues we have with IE on the web server)

I've tried all the resolutions posted for Error Message #2 here, including setting KeepAlive = False and ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3, however none have worked to fix the issue.

Our webserver is running Windows Server 2003 and IIS6.0

What can cause this error to occur on one machine, but not another?

Edit: Here's the Trace information:

System.Web.Services.Asmx Information: 0 : Calling XmlSerializer [Create XmlSerializer]
    Method: System.Xml.Serialization.XmlSerializer#1669721556::FromMappings(System.Xml.Serialization.XmlMapping[]#4002219=[4], ExternalWebService.ProcessRequestService#93032588=ExternalWebService.ProcessRequestService)
    Caller: System.Web.Services.Protocols.SoapClientType#42808772::.ctor(ExternalWebService.ProcessRequestService#93032588=ExternalWebService.ProcessRequestService)
    ProcessId=2796
    LogicalOperationStack=
    ThreadId=14
    DateTime=2012-12-21T17:02:39.2826250Z
    Timestamp=61755143048799345
System.Web.Services.Asmx Information: 0 : Return from XmlSerializer [Create XmlSerializer]
    Caller: System.Web.Services.Protocols.SoapClientType#42808772::.ctor(ExternalWebService.ProcessRequestService#93032588=ExternalWebService.ProcessRequestService)
    ProcessId=2796
    LogicalOperationStack=
    ThreadId=14
    DateTime=2012-12-21T17:02:39.6576250Z
    Timestamp=61755144323396760
System.Web.Services.Asmx Information: 0 : Calling XmlSerializer [Write Request]
    Method: Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer#1231684::Serialize(System.Xml.XmlTextWriter#8023888=.., System.Object[]#19695157=[40], (null), (null))
    Caller: ExternalWebService.ProcessRequestService#60553900::Serialize()
    ProcessId=2796
    LogicalOperationStack=
    ThreadId=14
    DateTime=2012-12-21T17:02:39.7357500Z
    Timestamp=61755144598320131
System.Web.Services.Asmx Information: 0 : Return from XmlSerializer [Write Request]
    Caller: ExternalWebService.ProcessRequestService#60553900::Serialize()
    ProcessId=2796
    LogicalOperationStack=
    ThreadId=14
    DateTime=2012-12-21T17:02:39.7357500Z
    Timestamp=61755144621621529
System.Web.Services.Asmx Information: 0 : Calling WebRequest.GetResponse
    Method: System.Net.HttpWebRequest#2800407::GetResponse()
    Caller: ExternalWebService.ProcessRequestService#60553900::GetWebResponse()
    ProcessId=2796
    LogicalOperationStack=
    ThreadId=14
    DateTime=2012-12-21T17:02:39.7357500Z
    Timestamp=61755144622114240

Answer 1

Apparently in my particular case, it was because the web service vendor had increased their security, and our web server didn't have the correct library needed to decipher the connection.

Our only clue was a single System event viewer entry with an ID of 36874 and a source of Schannel a few days back on the same day our site stopped working that said:

An SSL connection request was received from a remote client application, but none of the cipher suites supported by the client application are supported by the server. The SSL connection request has failed.

However if you're debugging this problem yourself, this link was very useful since it gave some simplified descriptions of the error which pointed me in the right direction, that my problem was related to not being able to connect to their web service.

The underlying connection was closed: An unexpected error occurred on a send

This problem occurs when the client computer cannot send an HTTP request. The client computer cannot send the HTTP request because the connection has been closed or is unavailable. Lookup the error in this article http://support.microsoft.com/kb/915599.

and the linked Microsoft support page was also fairly detailed about some different resolutions you could try to fix the most common causes of this problem.

Answer 2

Just an idea, don't know if it's the good one : the same thing happened few months ago on one of our server, because someone (security team ? :/) changed the write permission on the "%SYSTEMROOT%\Temp" folder. Since that action, XMLSerializer couldn't generate the temporary assembly used to serialize/deserialize.

From http://www.hanselman.com/blog/ChangingWhereXmlSerializerOutputsTemporaryAssemblies.aspx :

When using the XmlSerializer from ASP.NET there are permissions issues can can be solved by granting the user account read/write permissions on the %SYSTEMROOT%\Temp folder

So perhaps you can check that the account used by your application pool has write permissions on the "%SYSTEMROOT%\Temp" folder...

Good luck...