How do I troubleshoot PipeExceptions and CommunicationExceptions in WCF?

Question

I have an application consisting of several WCF services, some of which are implemented in Workflow Foundation (.NET 3.5), others just plain C#. These services communicate with each other over a netNamedPipeBinding for performance reasons. The trouble is that I'm seeing more and more CommunicationExceptions and underlying PipeExceptions as soon as the system load increases. The funny thing is that these transactions do seem to finish ultimately. One reason for that is that we have a retry mechanism in the workflows, but even the calls from plain C# services succeed even though I see these errors in the WCF traces. Is there some retry mechanism in the named pipe subsystem of Windows or something?

However I want these errors fixed, or at least understand the underlying problem. I feel they are impacting the performance and stability of the application. How do I go about and diagnose the root cause of these errors correctly if I don't see any other exception coming from the services themselves?

Here are some of the exceptions I get:

PipeException: There was an error reading from the pipe: The pipe has been ended. (109, 0x6d).

And:

PipeException: The operation cannot be completed because the pipe was closed. This may have been caused by the application on the other end of the pipe exiting.

within a TimeoutException: The pipe connection was aborted because an asynchronous read from the pipe did not complete within the allotted timeout of 00:02:00. The time allotted to this operation may have been a portion of a longer timeout.

Now the timeout exception seems to come from the fact that the system is having trouble handling the load. These operations are quite small normally but the amount of them seems to be the problem. Or could this the result of previous pipe connections being terminated and not given back to the pool?

I've tried experimenting with a serviceThrottling behavior in WCF config to increase the amount of instances etc. but these errors keep emerging. Any tips?

/EDIT: I did turn on both WCF tracing and message logging. That's where I'm seeing the PipeExceptions and CommunicationExceptions. The application itself is not showing any errors. We've instrumented the WCF services quite a bit so that all exceptions get logged using log4net and I don't see any errors in those logs at all. This all seems to be happening at WCF level.

Answer 1

You may want to activate logging and activity tracing on your service. Then use the Service Trace Viewer to have a clear view of the sequence of events.

This will give a huge amount of data to go through so I suggest you activate it just for one test case, and then deactivate it. This should limit the volume of data.