How does ServicePointManager.ReusePort and SO_REUSE_UNICASTPORT alleviate ephemeral port exhaustion?

Question

Windows 10 and Windows Server 2016 introduced the SO_REUSE_UNICASTPORT socket option. It was made available for use in .NET starting from version 4.6 via the ServicePointManager.ReusePort static property. I'm suffering from ephemeral port exhaustion on a .NET application during very high loads (many concurrent outgoing requests via HttpClient), and I am considering using this option to deal with it. I'm aware of other ways to deal with the problem (such as editing the Windows Registry to modify the max number of ephemeral ports or shortening TIME_WAIT), but I'd like to fully under this solution too.

The documentation for ServicePointManager.ReusePort is very minimal:

Setting this property value to true causes all outbound TCP connections from HttpWebRequest to use the native socket option SO_REUSE_UNICASTPORT on the socket. This causes the underlying outgoing ports to be shared. This is useful for scenarios where a large number of outgoing connections are made in a short time, and the app risks running out of ports.

Looking at the documentation for SO_REUSE_UNICASTPORT doesn't provide any additional insights:

When set, allow ephemeral port reuse for Winsock API connection functions which require an explicit bind, such as ConnectEx. Note that connection functions with an implicit bind (such as connect without an explicit bind) have this option set by default. Use this option instead of SO_PORT_SCALABILITY on platforms where both are available.

I couldn't find any explanation on the web as to how exactly this "ephemeral port reuse" is achieved, how exactly it works on a technical level, and how well it reduces the risk of ephemeral port exhaustion. How much of an improvement can I expect? Using this feature, how can I calculate a new limit for my application? Are there any downsides to enabling this?

This is all shrouded in mystery and I'd love it if someone can explain this new mechanism and its implications.

Answer 1

TCP connection is uniquely identified by (local IP, local port, remote IP, remote port). That means it's perfectly possible to use the same (local IP, local port) pair for multiple sockets connecting to different remote endpoints. Suppose you want to make http request to "site1.com" and "site2.com". You are using sockets with the following code:

using (var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp)) {                
    socket.Bind(new IPEndPoint(IPAddress.Parse("some local ip"), 45455));
    socket.Connect(server, port);
    socket.Send(someBytes);
    // ...
}

So you are binding socket to specific local endpoint with port 45455. If you now try to do that concurrently to make requests to "site1.com" and "site2.com" you will get "an address already in use" exception.

But if you add ReuseAddress option (note that it's not option your question is about) before binding:

socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);

You will be able to bind to sockets to the same local (ip, port) and you will see in netstat two ESTABLISHED connections.

All of the above is to show that in theory nothing prevents one to reuse even ephemeral port to make multiple connections to different remote endpoints. But, when you bind to ephemeral port (0) - it is not yet known to which remote endpoint you are going to connect. Suppose all ephemeral ports are in use and you are binding to 0. OS provides you some port for reuse during bind phase, there is one socket using this port already connected to "site1.com". You are trying to connect to "site1.com" too and that fails (because all 4 values identifying tcp connection are the same for both sockets).

What SO_REUSE_UNICASTPORT does is it delays choosing ephemeral port when you bind to 0 until actual connection phase (Connect() call for example). At this phase (unlike bind) you already know local ip, remote ip, remote port you are going to connect to, and you need to choose ephemeral port. Suppose all ports are in use. Now, instead of choosing some random port for reuse (and potentially fail later on connect) you can choose port which is connected to different remote endpoint (different from what current socket is trying to connect to).

You can confirm this for example at this MS support article:

SO_REUSE_UNICASTPORT

For a connection scenario to be implemented, the socket option must be set before binding a socket. This option instructs the system to postpone port allocation until connection time when the 4-tuple (quadruple) for the connection is known.

Note that SO_REUSE_UNICASTPORT only has effect on explicit bindings (as stated in your question quote, but still worth repeating). If you bind implicitly (such as when you just Connect() without binding) - this option is already set by default (where supported of course).

About which effect this has on your particular application. First from the above it should be clear that if your application makes ton of requests to the same remote endpoint (to the same http server for example) - this option will have no effect. If you make a lot of requests to different endpoints though - it should help to prevent ports exhaustion. Still effect of ServicePointManager.ReusePort itself depends on how HttpClient works with sockets internally I guess. If it just Connect() without explicit binding - this option should be enabled (on supported systems) by default, so setting ServicePointManager.ReusePort to true will not have additional effect, otherwise it will. Since you don't know (and should not rely on) its internal implementation its worth enabling ServicePointManager.ReusePort in your particular scenario.

You can also perform tests with this option on\off by limiting range of ephemeral ports (with command like netsh int ipv4 set dynamicport tcp) to some small amounts and see how it goes.