What could cause so many TIME_WAIT connections to be open?

Question

So, I have application A on one server which sends 710 HTTP POST messages per second to application B on another server, which is listening on a single port. The connections are not keep-alive; they are closed.

After a few minutes, application A reports that it can't open new connections to application B.

I am running netstat continuously on both machines, and see that a huge number of TIME_WAIT connections are open on each. Virtually all connections showing are in TIME_WAIT. From reading online, it seems that this is the state it's in for 30 seconds (on our machines 30 seconds according to /proc/sys/net/ipv4/tcp_fin_timeout value) after each side closes the connection.

I have a script running on each machine that's continuously doing:

netstat -na | grep 5774 | wc -l

and:

netstat -na | grep 5774 | grep "TIME_WAIT" | wc -l

The value of each, on each machine, seems to get to around 28,000 before application A reports that it can't open new connections to application B.

I've read that this file: /proc/sys/net/ipv4/ip_local_port_range provides the total number of connections that can be open at once:

$ cat /proc/sys/net/ipv4/ip_local_port_range 32768 61000

61000 - 32768 = 28232, which is right in line with the approximately 28,000 TIME_WAITs I am seeing.

My question is how is it possible to have so many connections in TIME_WAIT.

It seems that at 710 connections per second being closed, I should see approximately 710 * 30 seconds = 21300 of these at a given time. I suppose that just because there are 710 being opened per second doesn't mean that there are 710 being closed per second...

The only other thing I can think of is a slow OS getting around to closing the connections.

Answer 1

TCP's TIME_WAIT indicates that local endpoint (this side) has closed the connection. The connection is being kept around so that any delayed packets can be matched to the connection and handled appropriately. The connections will be removed when they time out within four minutes.

Assuming that all of those connections were valid, then everything is working correctly. You can eliminate the TIME_WAIT state by having the remote end close the connection or you can modify system parameters to increase recycling (though it can be dangerous to do so).

Vincent Bernat has an excellent article on TIME_WAIT and how to deal with it:

The Linux kernel documentation is not very helpful about what net.ipv4.tcp_tw_recycle does:

Enable fast recycling TIME-WAIT sockets. Default value is 0. It should not be changed without advice/request of technical experts.

Its sibling, net.ipv4.tcp_tw_reuse is a little bit more documented but the language is about the same:

Allow to reuse TIME-WAIT sockets for new connections when it is safe from protocol viewpoint. Default value is 0. It should not be changed without advice/request of technical experts.

The mere result of this lack of documentation is that we find numerous tuning guides advising to set both these settings to 1 to reduce the number of entries in the TIME-WAIT state. However, as stated by tcp(7) manual page, the net.ipv4.tcp_tw_recycle option is quite problematic for public-facing servers as it won’t handle connections from two different computers behind the same NAT device, which is a problem hard to detect and waiting to bite you:

Enable fast recycling of TIME-WAIT sockets. Enabling this option is not recommended since this causes problems when working with NAT (Network Address Translation).