WCF "Self-Hosted" application becomes unresponsive

Question

We have a C# (.Net 4.0) console application that "self hosts" two WCFs services: one used WSHttpBinding, and another uses BasicHttpBinding.

Connecting to these services, we have two separate client applications: a Silverlight-based service that uses the BasicHttpBinding, and another console app that uses the WSHttpBinding.

The WCF service app usually has around 30 users connected via the Silverlight client, and another couple of connections from the console application client. It's not "flat out" by any means; each client queries the WCF service maybe once every 5 seconds at the very most.

The problem is: intermittently the service application becomes unresponsive. Although the server itself continues to run (it continues to write to a log file), all WCF activity (on both ServiceHosts) appear to "seize". New requests aren't processed (although the TCP connections are accepted). Also, the number of threads consumed by the application starts to ramp up dramatically, at the rate of around one new thread per second. The code itself doesn't do anything with Threads or ThreadPools, although it occasionally will issue a Thread.Sleep for a few hundred milliseconds.

The frustrating thing is the intermittent nature of the problem: the code regularly runs for hours, even days without any issues. Then, with no apparent cause, it suddenly becomes unresponsive and the Thread count starts ticking up.

I've tried simulating user activity - connecting and disconnecting clients, "swamping" the service with requests - but I can do nothing to reproduce the fault.

Just in case the issue was WCF Throttling, I've added this code:

 ServiceThrottlingBehavior throttlingBehavior = new System.ServiceModel.Description.ServiceThrottlingBehavior
                                                           {
                                                               MaxConcurrentCalls = 512,
                                                               MaxConcurrentInstances = 8192,
                                                               MaxConcurrentSessions = 8192
                                                           };

        host.Description.Behaviors.Add(throttlingBehavior);
        host2.Description.Behaviors.Add(throttlingBehavior);

.. with no apparent effect.

I've put extensive logging in the code to try and determine what it is that triggers this behaviour - logging each call to each method - but nothing has appeared as a result. I've wrapped everything in try...catch blocks and spitting any exceptions to the log file, to see if something's falling over somewhere, and also trapped the UnhandledExceptions in a similar fashion ... but again, nothing appears to be going wrong.

Does the above behaviour sound familiar to anyone, or can anyone suggest on the best way forward to troubleshoot this issue?

EDIT: following Wal's advice below, I've captured a .DMP of the application when it starts mis-behaving, and looking at the Parallel Stacks view in VS2012, I see:

... and others very similar but with different numbers of threads. I'm not clever enough to decode exactly what this means.. can anyone suggest where to start looking next?

Answer 1

what is the concurrencymode for the service? and the instancecontextmode?

the default instancecontextmode is per session, it may be worth changing this to percall, this will use more memory but will ensure that each service instance is no hanging around (provided the client is correctly disposed of http://coding.abel.nu/2012/02/using-and-disposing-of-wcf-clients/)

Answer 2

As pointed out sooner, sounds like you have a race condition. Aren't you by any chance checking System.ServiceModel.ICommunicationObject.State of the connection somewhere in the code? See MSDN article :

Checking the value of the System.ServiceModel.ICommunicationObject.State property is 
a race condition and is not recommended to determine whether to reuse or close a channel.