TCP Socket Server Builds Up CLOSE_WAITs Occasionally Over Time Until Inoperable

Question

Hopefully someone can help us as we're reaching as far as investigation can go!

We've got a simple asynchronous socket server written in C# that accepts connections from an ASP.NET web application, is sent a message, performs some processing (usually against a DB but other systems too) and then sends a response back to the client. The client is in charge of closing the connection.

We've been having issues where if the system is under heavy load over a long period of time (days usually), CLOSE_WAIT sockets build up on the server box (netstat -a) to an extent that the process will not accept any further connections. At that point we have to bounce the process and off it runs again.

We've tried running some load tests of our ASP.NET application to attempt to replicate the problem (because inferring some issue from the code wasn't possible). We think we've managed this and ended up with a WireShark packet trace of the issue manifesting itself as a SocketException in the socket server's logs:

System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.BeginSend(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)

I've tried to reproduce the issue from the packet trace as a single threaded process directly talking to the socket server (using the same code the ASP.NET app does) and am unable.

Has anybody got any suggestions of next things to try, check for or obvious things we may be doing wrong?

Answer 1

Look at the diagram

http://en.wikipedia.org/wiki/File:Tcp_state_diagram_fixed.svg

Your client closed the connection by calling close(), which sent FIN to the server socket, which ACKed the FIN and the state of which now changed to CLOSE_WAIT, and stays that way unless the server issues close() call on that socket.

Your server program needs to detect whether the client has aborted the connection, and then close() it immediately to free up the port. How? Refer to read(). Upon reading end-of-file (meaning FIN is received), zero is returned.