WCF timing out too soon & 'Connection was closed unexpectedly' exception

Question

A WCF client returns the above exception after 120 seconds when calling a method which returns a string "success". Logs from the server show it executes fully and completes in around 3 minutes. With tracing enabled I see no errors thrown. Any help would be greatly appreciated as other suggestions from similar problems unfortunately haven't helped. Thanks!

EDIT - Solved: Turns out that the client was using a Smoothwall filtering device which was closing connections after two minutes.

Exception:

System.ServiceModel.CommunicationException The underlying connection was closed: The connection was closed unexpectedly.
Stack Trace:    at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)

InnerException:

System.Net.WebException The underlying connection was closed: The connection was closed unexpectedly.The underlying connection was closed: The connection was closed unexpectedly.
Stack Trace:    at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)

As far as I can see, the bindings are set up correctly and the timeouts are beyond 120 seconds. Here's the client binding:

<bindings>
  <customBinding>
      <binding name="CustomBinding_IPHG" closeTimeout="00:05:00"
          openTimeout="00:05:00" receiveTimeout="00:15:00" sendTimeout="00:15:00">
          <security authenticationMode="SecureConversation" requireDerivedKeys="false">
              <localClientSettings maxClockSkew="00:30:00" />
              <localServiceSettings maxClockSkew="00:30:00" />
              <secureConversationBootstrap authenticationMode="UserNameOverTransport">
                  <localClientSettings maxClockSkew="00:30:00" />
                  <localServiceSettings maxClockSkew="00:30:00" />
              </secureConversationBootstrap>
          </security>
          <textMessageEncoding>
              <readerQuotas maxStringContentLength="5242880" />
          </textMessageEncoding>
          <httpsTransport maxReceivedMessageSize="6553600" useDefaultWebProxy="true" />
      </binding>
  </customBinding>
</bindings>
<behaviors>
<endpointBehaviors>
<behavior name="commonBehaviour">
  <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
  <clientCredentials>
    <serviceCertificate>
      <authentication certificateValidationMode="PeerOrChainTrust"/>
    </serviceCertificate>
  </clientCredentials>
</behavior>
</endpointBehaviors>
</behaviors>

Here's the server binding:

<bindings>
  <customBinding>
    <binding name="CommonBinding" closeTimeout="00:05:00" openTimeout="00:05:00" receiveTimeout="00:15:00" sendTimeout="00:15:00">
      <transactionFlow />
      <security authenticationMode="SecureConversation" requireDerivedKeys="false">
        <secureConversationBootstrap authenticationMode="UserNameOverTransport">
          <localServiceSettings maxClockSkew="00:30:00" />
          <localClientSettings maxClockSkew="00:30:00" />
        </secureConversationBootstrap>
        <localServiceSettings maxClockSkew="00:30:00" />
        <localClientSettings maxClockSkew="00:30:00" />
      </security>
      <textMessageEncoding>
        <readerQuotas maxStringContentLength="41943040"/>
      </textMessageEncoding>
      <httpsTransport maxBufferSize="41943040" maxReceivedMessageSize="41943040" />
    </binding>
  </customBinding>
</bindings>
<behaviors>
  <serviceBehaviors>
    <behavior name="BehaviorRBS">
      <serviceMetadata httpsGetEnabled="true" httpGetEnabled="false" />
      <serviceDebug includeExceptionDetailInFaults="true" httpHelpPageEnabled="false" />
      <dataContractSerializer maxItemsInObjectGraph="2147483647" />
      <serviceCredentials>
        <userNameAuthentication userNamePasswordValidationMode="Custom" customUserNamePasswordValidatorType="PHGAPI.PHGUserNamePassValidator, PHGAPI"/>
        <serviceCertificate
                findValue="*.domainname.com"
                x509FindType="FindBySubjectName"
                storeLocation="LocalMachine"
                storeName="My" />
      </serviceCredentials>
      <serviceThrottling maxConcurrentCalls="200" maxConcurrentSessions="400" maxConcurrentInstances="600" />
    </behavior>
  </serviceBehaviors>
</behaviors>
<services>
    <service behaviorConfiguration="BehaviorRBS" name="PHGAPI.iAPI">
      <endpoint address="" binding="customBinding" bindingConfiguration="CommonBinding" contract="PHGAPI.IiAPI" />
    </service>
</services>

This is the interface for the method in question:

[OperationContract]
string SetSuspensions(List<Suspension> suspensions);

Answer 1

I would look into WCF and related configuration settings that have default values of 120 seconds. To that end, you may want to look into the HTTP.sys connectionTimeout value.

From the Microsoft documentation:
Connections can be considered inactive for the following reasons:

• The HTTP.sys Timer_ConnectionIdle timer expired. The connection expired and remains idle.
• The HTTP.sys Timer_EntityBody timer expired. The connection expired before the request entity body arrived. When it is clear that a request has an entity body, the HTTP API turns on the Timer_EntityBody timer. Initially, the limit of this timer is set to the connectionTimeout value. Each time another data indication is received on this request, the HTTP API resets the timer to give the connection more minutes as specified in the connectionTimeout attribute.
• The HTTP.sys Timer_AppPool timer expired. The connection expired because a request waited too long in an application pool queue for a server application to dequeue and process it. This time-out duration is connectionTimeout.

The default value is 00:02:00 (two minutes).

http://www.iis.net/configreference/system.applicationhost/sites/sitedefaults/limits

Good luck.

Answer 2

Tracing these issues can be a pain. Try the following to help track the problem down.

1) Identify where the problem is i.e. client or server. Start your Service in debug mode in the IDE and then use the WCF Test client rather than your own client to rule out client configuration problems.

2) Install and use Fiddler2 to inspect the HTTP messages going to\coming from the service. You will see if a response is coming back from the service and what's in it.

3) Assuming this is a synchronous call, holding a thread open for three minutes is bad for performance anyhow. Perhaps look into refactoring the service to either return the data a lot quicker or make it async .

Answer 3

Based on the information posted in your questions and the comments below, it sounds to me like its a timeout on the client section of this equation. Since the default timeout on the generated client is pretty small. Simply have your client modify their config to increase the timeout.

--exert (client config)--

<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
    </startup>
    <system.serviceModel>
        <bindings>
            <basicHttpBinding>
                <binding name="BasicHttpBinding_IService1" closeTimeout="00:10:00" openTimeout="00:10:00" receiveTimeout="00:10:00" sendTimeout="00:10:00"  />
            </basicHttpBinding>
        </bindings>
        <client>
            <endpoint address="http://localhost:8733/Design_Time_Addresses/WcfServiceLibrary1/Service1/"
                binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_IService1"
                contract="ServiceReference1.IService1" name="BasicHttpBinding_IService1" />
        </client>
    </system.serviceModel>
</configuration>

The bindings you are using are different, but all bindings have these properties. I usually increase all the timeouts, but the one thats going to be most important for you is the "receiveTimeout"

Answer 4

Where is the client located? (Same company network, same country)

what is the network speed at client side?

What is the size in bytes of the parameter List suspensions ?

Did you try to use Fiddler or some other network package software to capture the transmission?

In a prior job there was a case where every error in the server was logged to a different server; the issue was that an incoming message with 10,000 bad rows was sent to the log but not as a single failed error but as 10,000 times the same error.

Serializing this huge 200MB+ object to be sent to the other server was causing the internal network to be seen as 100% utilized and making the server to go 100%, thus IIS was not able to respond on time to any other request.

I'm not saying this may be your case, I'm just sharing a timeout situation I had where the problem was not in the method that timed out but in another component in the services.