High performance C# TCP server problem: No connection could be made because the target machine actively refused it

Question

I have developed a TCP server according to your advises: High performance TCP server in C#

It is based on asynchron pattern.

I also developed a stress test application to test its performance. My server can get thousands of connections paralelly from my stress test app, can parse data and save it to my database.

When I stress my server, I can get "System.Net.Sockets.SocketException "No connection could be made because the target machine actively refused it" error from my server, so I have to reconnect to it. If I test it with 5000 concurrent connections, I have to try connect again because of this problem 10-20% of the connections, if I test it with 10K concurrent connections, it can be 30-40%. Sometimes it can be - very rarely - more, than 50%. It seems it can not handle connection accepts: I make new connections from my stress test as heavily as my test machine can - about 120 connections/sec.

So, what can cause this kind of exception? How to handle it? What to do in server side implementation to avoid this problem? How to tune TCP connection accept?

Thanks in advance!

Answer 1

You might be running out of available ports every now and then. You can view this easily using SysInternals' TcpView utility.

On Windows, when you release a port, it doesn't immediately go into an available state, but instead sits in a TIME_WAIT state for some interval. Until it leaves this state, no app can use this port. The time delay, the max number of ports, and the available port ranges are all different to the OS, XP vs Win7 vs Win2008 Server.

There are two registry entries that can reduce this time interval: HKLM/System/CurrentControlSet/Services/Tcpip/Parameters/TCPTimedWaitDelay

and increase the max number of ports that can be opened by an app: HKLM/System/CurrentControlSet/Services/Tcpip/Parameters/MaxUserPort

EDIT: MaxFreeTcbs seems to be a third setting which could help (I haven't tried this yet), mentioned in this TechNet article which has more advice on tracking down odd network problems. HTH.