How do I prevent Socket/Port Exhaustion?

Question

I am attempting to performance test a website by hitting it with requests across multiple threads. Each thread executes n times. (in a for loop)

However, I am running into problems. Specifically the WebException ("Unable to connect to remote server") with the inner exception:

An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full 127.0.0.1:52395

I am attempting to run 100 threads at 500 iterations per thread.

Initially I was using HttpWebRequest in System.Net to make the GET request to the server. Currently I am using WebClient as I assumed that each iteration was using a new socket (so 100 * 500 sockets in a short period of time). I assumed WebClient (which is instantiated once per thread) would only use one socket.

I don't need 50 000 sockets open at once, as I would like to send the GET request, receive the response, and close the socket, freeing it for use in the next loop iteration. I understand that it would be a problem to

However, even with WebClient, a bunch of sockets are being requested resulting in a bunch of sockets in TIME_WAIT mode (checked using netstat). This causes other applications (like internet browsers) to hang and stop functioning.

I can operate my test with less iterations and/or less threads, as it appears the sockets do eventually exit this TIME_WAIT state. However, this is not a solution as it doesn't adequately test the abilities of the web server.

Question:

How do I explicitly close a socket (from the client side) after each thread iteration in order to prevent TIME_WAIT states and socket exhaustion?

Code:

Class that wraps the HttpRequest

Edit: Wrapped WebClient in a using, so a new one is instantiated,used and disposed for every iteration. The problem still persists.

  public sealed class HttpGetTest : ITest {
    private readonly string m_url;

    public HttpGetTest( string url ) {          
        m_url = url;
    }

    void ITest.Execute() {
        using (WebClient webClient = new WebClient()){
            using( Stream stream = webClient.OpenRead( m_url ) ) {          
            }
        }
    }
}

The part of my ThreadWrapperClass that creates a new thread:

public void Execute() {
    Action Hammer = () => {
        for( int i = 1; i <= m_iterations; i++ ) {
            //Where m_test is an ITest injected through constructor
            m_test.Execute();
        }       
    };
    ThreadStart work = delegate {
        Hammer();
    };
    Thread thread = new Thread( work );
    thread.Start();
}

Answer 1

Do you understand the purpose of TIME_WAIT? It's a period during which it would be unsafe to reuse the port because lost packets (that have been successfully retransmitted) from the previous transaction might yet be delivered within that time period.

You could probably tweak it down in the registry somewhere, but I question if this is a sensible next step.

My experience of creating realistic load in a test environment have proved very frustrating. Certainly running your load-tester from localhost is by no means realistic, and most network tests I have made using the .net http apis seem to require more grunt in the client than the server itself.

As such, it's better to move to a second machine for generating load on your server... however domestic routing equipment is rarely up to the job of supporting anywhere near the number of connections that would cause any sort of load on a well written server app, so now you need to upgrade your routing/switching equipment as well!

Lastly, I've had some really strange and unexpected performance issues around the .net Http client API. At the end of the day, they all use HttpWebRequest to do the heavy lifting. IMO it's nowhere near as performant as it could be. DNS is sychronous, even when calling the APIs asynchronously (although if you're only requesting from a single host, this isn't an issue), and after sustained usage CPU usage creeps up until the client becomes CPU constrained rather than IO constrained. If you're looking to generate sustained and heavy load, any request-heavy app reliant on HttpWebRequest is IMO a bogus investment.

All in all, a pretty tricky job, and ultimately, something that can only be proved in the wild, unless you've got plently of cash to spend on an armada of better equipment.

[Hint: I got much better perfomance from my own client written using async Socket apis and a 3rd party DNS client library]

Answer 2

Q: How do I explicitly close a socket ... in order to prevent TIME_WAIT states?

A: Dude, TIME_WAIT is an integral - and important! - part of TCP/IP itself!

You can tune the OS to reduce TIME_WAIT (which can have negative repercussions).

And you can tune the OS to increase #/ephemeral ports:

• http://msdn.microsoft.com/en-us/library/aa560610%28v=bts.20%29.aspx

Here's a link on why TIME_WAIT exists ... and why it's a Good Thing:

• http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html

Answer 3

It's not an issue of closing sockets or releasing resources in your app. The TIME _WAIT is a TCP stack timeot on released sockets to prevent their re-use until such time as it is virtually impossible for any packets 'left over' from a previous connection to that socket to not have expired.

For test purposes, you can reduce the wait time from the default, (some minutes, AFAIK), to a smaller value. When load-testing servers, I set it at six seconds.

It's in the registry somewhere - you'll find it if you Google.

Found it:

Change TIME_WAIT delay

Answer 4

It looks like you are not forcing your WebClient to get rid of the resources that it has allocated. You are performing a Using on the stream that is returned, but your WebClient still has resources.

Either wrap your WebClient instantiation in a using block, or manually call dispose on it once you are done reading from the URL.

Try this:

public sealed class HttpGetTest : ITest {
    private readonly string m_url;

    public HttpGetTest( string url ) {
        m_url = url;        
    }

    public void ITest.Execute() {
        using( var m_webClient = new WebClient())
        {
            using( Stream stream = m_webClient.OpenRead( m_url ) ) 
            {

            }
        }
    }
}