WCF: The socket connection was aborted. A receive timeout being exceeded by the remote host

Question

I have been working through a head-scratcher with WCF. I have a WCF Service (hosted as a Windows Service) that is called by a console application. It works great, but recently we have run into a timeout issue at 10 minutes when we had to run a query against a legacy system.

I use log4net in both the service and the client .exe, and what is strange is that the service actually completes the job and throws no exception. It runs the query (which takes about 12 minutes, and creates a file, and logs success).

The console application, however, logs an exception, claiming that the "host" returned a timeout or an error. I presume it's a timeout and not a buffer issue, because the exception always happens exactly 10 minutes to the second after the call is initially made.

Here is the exception/stack info from my log4net:

2015-07-28 17:35:47,364 [1] 
System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:29:59.9843999'. ---> System.IO.IOException: The read operation failed, see inner exception. ---> System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:29:59.9843999'. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)
   --- End of inner exception stack trace ---
   at System.ServiceModel.Channels.SocketConnection.ReadCore(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, Boolean closing)
   at System.ServiceModel.Channels.SocketConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at System.ServiceModel.Channels.DelegatingConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at System.ServiceModel.Channels.ConnectionStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Security.NegotiateStream.StartFrameHeader(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.NegotiateStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   --- End of inner exception stack trace ---
   at System.Net.Security.NegotiateStream.ProcessRead(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.NegotiateStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.ServiceModel.Channels.StreamConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   --- End of inner exception stack trace ---

Server stack trace: 
   at System.ServiceModel.Channels.StreamConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout)
   at System.ServiceModel.Channels.SessionConnectionReader.Receive(TimeSpan timeout)
   at System.ServiceModel.Channels.SynchronizedMessageSource.Receive(TimeSpan timeout)
   at System.ServiceModel.Channels.TransportDuplexSessionChannel.Receive(TimeSpan timeout)
   at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceive(TimeSpan timeout, Message& message)
   at System.ServiceModel.Dispatcher.DuplexChannelBinder.Request(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: 
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at FileGeneratorConsole.FileGeneratorServiceReference.IFileGeneratorService.GenerateXmlFileFromSql(String sqlServer, String sqlDatabase, String sqlQuery, Int32 commandTimeout, Boolean allowEmptyResult, String outputFileName, String xslFileName, Boolean archiveExistingFile)
   at FileGeneratorConsole.Program.Main(String[] args)

Now, I thought I had done my research and figured out what the issue could be. I assumed it was the sendTimeout and receiveTimeout that are part of the binding definition, so I've added that:

<bindings>
      <netTcpBinding>
        <binding name="NetTcpBinding_IFileGeneratorService"
                  maxBufferPoolSize="2147483647"
                 maxBufferSize="2147483647"
                 maxReceivedMessageSize="2147483647"
                 receiveTimeout="00:30:00"
                 closeTimeout="00:30:00"
                 openTimeout="00:30:00"
                 sendTimeout="00:30:00">
        </binding>
      </netTcpBinding>
    </bindings>

And, if you notice in the exception above, you'll see that the error says:

Local socket timeout was '00:29:59.9843999'

It used to say 00:09:59.... so it looks like that took.

I also added this to the service config file, just in case the binding needed to be on both sides, but that didn't help.

Lastly, out of desperation, I added a few other things. I tried adding timeouts to the service section:

and I tried adding

to the behavior itself.

I'm at a loss now, it always dies at 10 minutes.

Thanks in advance

Answer 1

It's always simpler than you think.

Sadly, what was missing all this time was the bindingConfiguration reference in the service.endpoint. All this time, the service was running assumedly using the default binding definition. I had a binding defined in my config, but had missed the part where you specify:

I had:

 <endpoint address="" binding="netTcpBinding" bindingConfiguration="">

I needed:

 <endpoint address="" binding="netTcpBinding" bindingConfiguration="NetTcpBinding_IFileGeneratorService">

Once that was in place, my actual service endpoint was looking at the right binding properties.