Thread starvation on net.tcp binding - TCP error code 10061

Question

I have faced a very strange error in my WCF service, which appears to somehow create a deadlock or thread starvation in socket level when I use NetTcpBinding. I have a quite simple self-hosted service:

class Program
{
    static void Main(string[] args)
    {
        using (ServiceHost serviceHost = new ServiceHost(typeof(TestService)))
        {
            serviceHost.Open();             
            Console.WriteLine("Press <ENTER> to terminate service.");
            Console.ReadLine();
            serviceHost.Close();
        }
        Uri baseAddress = new Uri("net.tcp://localhost:8014/TestService.svc");         
    }
}

[ServiceContract]
public interface ITestService
{
    [OperationContract]
    string GetData(string data);
}

public class TestService: ITestService
{
    public string GetData(string data)
    {
        Console.WriteLine(data);
        Thread.Sleep(5000);
        return "Ok";
    }
}

The configuration part:

<system.serviceModel>
<bindings>
  <basicHttpBinding>
    <binding name="basicHttpBinding" closeTimeout="00:02:00" openTimeout="00:02:00"
      receiveTimeout="00:02:00" sendTimeout="00:02:00" maxBufferSize="2000000000"
      maxReceivedMessageSize="2000000000" />
  </basicHttpBinding>
  <netTcpBinding>
    <binding name="netTcpBinding" closeTimeout="00:02:00" openTimeout="00:02:00"
      receiveTimeout="00:02:00" sendTimeout="00:02:00" listenBacklog="2000"
      maxBufferSize="2000000000" maxConnections="1000" maxReceivedMessageSize="2000000000">
      <security mode="None">
        <transport protectionLevel="EncryptAndSign" />
      </security>
    </binding>
    <binding name="TestServiceTcpEndPoint">
      <security mode="None" />
    </binding>
  </netTcpBinding>      
</bindings>

<behaviors>
  <serviceBehaviors>
    <behavior name="CommonServiceBehavior">
      <serviceMetadata httpGetEnabled="true" />
      <serviceDebug includeExceptionDetailInFaults="true" />
      <serviceThrottling maxConcurrentCalls="1000" maxConcurrentSessions="1000" maxConcurrentInstances="1000" />
    </behavior>
  </serviceBehaviors>
</behaviors>
<services>
  <service name="ServiceLauncher.TestService" behaviorConfiguration="CommonServiceBehavior">
    <endpoint address="" binding="netTcpBinding" bindingConfiguration="netTcpBinding" name="TestServiceTcpEndPoint" contract="ServiceLauncher.ITestService" />
    <endpoint address="" binding="basicHttpBinding" bindingConfiguration="basicHttpBinding" name="TestServiceTcpEndPoint" contract="ServiceLauncher.ITestService" />
    <endpoint address="mex"  binding="mexHttpBinding" bindingName="mexHttpBinding" contract="IMetadataExchange" />
    <host>
      <baseAddresses>
        <add baseAddress="net.tcp://localhost:8014/TestService.svc"/>
        <add baseAddress="http://localhost:1234/TestService.svc"/>
      </baseAddresses>
    </host>
  </service>
</services>
</system.serviceModel>

And I have a client which consumes this service in many threads with creating new instance for every thread (it is a requirement):

    static void Main(string[] args)
    {           
        for (int i = 0; i < 1000; i++)
        {
            Thread tr = new Thread(() =>
            {
                using (var service = new Test.TestServiceClient())
                {
                    var result = service.GetData(i.ToString());
                    Console.WriteLine(string.Format("{0}: {1} {2}",
                                      DateTime.Now,
                                      result,
                                      Thread.CurrentThread.ManagedThreadId));
                }  
            });
            tr.Start();                
        }
        Console.ReadLine();       
    }

In this case after some requests client raises EndpointNotFoundException, TCP error code 10061, No connection could be made because the target machine actively refused it. The number of requests is different all the time, and it is not the server part because it still works in normal state. And I see it keeps recieving the requests, what is most strangest in this situation. What is also strange that it can make your client host "immortal" after the exception - so that you can't kill it by any mean, except of the reboot of the system. I'm pretty sure that the problem is in low socket level of the client, and it is somehow connected with such a large number of threads, but I didn't succeed in finding something which could explaine the problem.

Answer 1

Every time I've seen the error "No connection could be made because the target machine actively refused it." the problem has not been with the service. Its usually a problem reaching the service.

A couple suggestions:

1. Avoid using with WCF Proxies. You can pick from several reasonable work arounds.

2. Read my answer to WCF performance, latency and scalability. Other than starting threads the old fashioned way, its basically the same test app. The post describes all the client causes (I could find) that cause “No connection could be made because the target machine actively refused it” and offers different WCF, TCP, and thread pool settings that can be adjusted.