Proper order for closing TcpListener and TcpClient connections (which side should be the active close)

Question

I read this answer on a previous question which says:

So the peer that initiates the termination – i.e. calls close() first – will end up in the TIME_WAIT state. [...]

However, it can be a problem with lots of sockets in TIME_WAIT state on a server as it could eventually prevent new connections from being accepted. [...]

Instead, design your application protocol so the connection termination is always initiated from the client side. If the client always knows when it has read all remaining data it can initiate the termination sequence. As an example, a browser knows from the Content-Length HTTP header when it has read all data and can initiate the close. (I know that in HTTP 1.1 it will keep it open for a while for a possible reuse, and then close it.)

I'd like to implement this using TcpClient/TcpListener, but it's not clear how to make it work properly.

Approach 1: both sides close

This is the typical way most MSDN examples illustrate - both sides calling Close(), not just the client:

private static void AcceptLoop()
{
    listener.BeginAcceptTcpClient(ar =>
    {
        var tcpClient = listener.EndAcceptTcpClient(ar);

        ThreadPool.QueueUserWorkItem(delegate
        {
            var stream = tcpClient.GetStream();
            ReadSomeData(stream);
            WriteSomeData(stream);
            tcpClient.Close();   <---- note
        });

        AcceptLoop();
    }, null);
}

private static void ExecuteClient()
{
    using (var client = new TcpClient())
    {
        client.Connect("localhost", 8012);

        using (var stream = client.GetStream())
        {
            WriteSomeData(stream);
            ReadSomeData(stream);
        }
    }
}

After I run this with 20 clients, TCPView shows a lot of sockets from both the client and server stuck in TIME_WAIT, which take quite some time to disappear.

Approach 2: just client close

As per the quotes above, I removed the Close() calls on my listener, and now I just rely on the client closing:

var tcpClient = listener.EndAcceptTcpClient(ar);

ThreadPool.QueueUserWorkItem(delegate
{
    var stream = tcpClient.GetStream();
    ReadSomeData(stream);
    WriteSomeData(stream);
    // tcpClient.Close();   <-- Let the client close
});

AcceptLoop();

Now I no longer have any TIME_WAIT, but I do get sockets left in various stages of CLOSE_WAIT, FIN_WAIT, etc. that also take a very long time to disappear.

Approach 3: give client time to close first

This time I added a delay before closing the server connection:

var tcpClient = listener.EndAcceptTcpClient(ar);

ThreadPool.QueueUserWorkItem(delegate
{
    var stream = tcpClient.GetStream();
    ReadSomeData(stream);
    WriteSomeData(stream);
    Thread.Sleep(100);      // <-- Give the client the opportunity to close first
    tcpClient.Close();      // <-- Now server closes
});

AcceptLoop();

This seems to be better - now only client sockets are in TIME_WAIT; the server sockets have all closed properly:

This seems to agree with what the previously linked article says:

So the peer that initiates the termination – i.e. calls close() first – will end up in the TIME_WAIT state.

Questions:

1. Which of these approaches is the right way to go, and why? (Assuming I want the client to be the 'active close' side)
2. Is there a better way to implement approach 3? We want the close to be initiated by the client (so that the client is left with the TIME_WAITs), but when the client closes, we also want to close the connection on the server.
3. My scenario is actually opposite to a web server; I have a single client that connects and disconnects from many different remote machines. I'd rather the server have connections stuck in TIME_WAIT instead, to free up resources on my client. In this case, should I make the server perform the active close, and put the sleep/close on my client?

Full code to try it yourself is here:

https://gist.github.com/PaulStovell/a58cd48a5c6b14885cf3

Edit: another useful resource:

http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html

For a server that does establish outbound connections as well as accepting inbound connections then the golden rule is to always ensure that if a TIME_WAIT needs to occur that it ends up on the other peer and not the server. The best way to do this is to never initiate an active close from the server, no matter what the reason. If your peer times out, abort the connection with an RST rather than closing it. If your peer sends invalid data, abort the connection, etc. The idea being that if your server never initiates an active close it can never accumulate TIME_WAIT sockets and therefore will never suffer from the scalability problems that they cause. Although it's easy to see how you can abort connections when error situations occur what about normal connection termination? Ideally you should design into your protocol a way for the server to tell the client that it should disconnect, rather than simply having the server instigate an active close. So if the server needs to terminate a connection the server sends an application level "we're done" message which the client takes as a reason to close the connection. If the client fails to close the connection in a reasonable time then the server aborts the connection.

On the client things are slightly more complicated, after all, someone has to initiate an active close to terminate a TCP connection cleanly, and if it's the client then that's where the TIME_WAIT will end up. However, having the TIME_WAIT end up on the client has several advantages. Firstly if, for some reason, the client ends up with connectivity issues due to the accumulation of sockets in TIME_WAIT it's just one client. Other clients will not be affected. Secondly, it's inefficient to rapidly open and close TCP connections to the same server so it makes sense beyond the issue of TIME_WAIT to try and maintain connections for longer periods of time rather than shorter periods of time. Don't design a protocol whereby a client connects to the server every minute and does so by opening a new connection. Instead use a persistent connection design and only reconnect when the connection fails, if intermediary routers refuse to keep the connection open without data flow then you could either implement an application level ping, use TCP keep alive or just accept that the router is resetting your connection; the good thing being that you're not accumulating TIME_WAIT sockets. If the work that you do on a connection is naturally short lived then consider some form of "connection pooling" design whereby the connection is kept open and reused. Finally, if you absolutely must open and close connections rapidly from a client to the same server then perhaps you could design an application level shutdown sequence that you can use and then follow this with an abortive close. Your client could send an "I'm done" message, your server could then send a "goodbye" message and the client could then abort the connection.

Answer 1

This is just how TCP works, you can't avoid it. You can set different timeouts for the TIME_WAIT or FIN_WAIT on your server, but that's about it.

The reason for this is that on TCP, a packet can arrive to a socket you've closed a long time ago. If you already have another socket open on the same IP and port, it would receive data meant for the previous session, which would confuse the hell out of it. Especially given that most people consider TCP to be reliable :)

If both your client and server implement TCP properly (for example, handling the clean shutdown correctly), it doesn't really matter whether the client or the server closed the connection. Since it sounds like you manage both sides, it shouldn't be an issue.

Your issue seems to be with the proper shutdown on the server. When one side of the socket closes, the other side will Read with a length of 0 - that's your message that the communication is over. You're most likely ignoring this in your server code - it's a special case that says "you can now safely dispose of this socket, do it now".

In your case, a server-side close seems like the best fit.

But really, TCP is rather complicated. It doesn't help that most of the samples on the internet are severely flawed (especially the samples for C# - it's not too hard to find a good sample for C++, for example) and ignore many of the important parts of the protocol. I've got a simple sample of something that might work for you - https://github.com/Luaancz/Networking/tree/master/Networking%20Part%201 It still isn't perfect TCP, but it's much better than the MSDN samples, for example.