Max number of concurrent HttpWebRequests

Question

I'm stress testing a web app and have set up a windows test program that spins up a number of threads and sends out a web request on each one.

Problem is I get the following output:

01/09/09 11:34:04 Starting new HTTP request on 10 01/09/09 11:34:04 Starting new HTTP request on 11 01/09/09 11:34:04 Starting new HTTP request on 13 01/09/09 11:34:05 Starting new HTTP request on 14 01/09/09 11:34:05 Starting new HTTP request on 11 01/09/09 11:34:05 11 has finished! 01/09/09 11:34:05 Starting new HTTP request on 13 01/09/09 11:34:05 13 has finished! 01/09/09 11:34:05 Starting new HTTP request on 14 01/09/09 11:34:05 14 has finished! 01/09/09 11:34:05 Starting new HTTP request on 11 01/09/09 11:34:05 11 has finished! 01/09/09 11:34:05 Starting new HTTP request on 14 01/09/09 11:34:05 14 has finished! 01/09/09 11:34:05 Starting new HTTP request on 13 01/09/09 11:34:05 13 has finished! 01/09/09 11:34:05 Starting new HTTP request on 15 01/09/09 11:34:06 Starting new HTTP request on 11 01/09/09 11:34:06 11 has finished! 01/09/09 11:34:06 Starting new HTTP request on 14 01/09/09 11:34:06 14 has finished! 

which sort of looks like there's a maximum of 5 threads, even if I create 100 as so:

int numberOfThreads = Convert.ToInt32(txtConcurrentThreads.Text);      List<BackgroundWorker> workers = new List<BackgroundWorker>();      for (int N = 0; N < numberOfThreads; N++)     {          BackgroundWorker worker = new BackgroundWorker();         worker.DoWork += new DoWorkEventHandler(worker_DoWork);         worker.RunWorkerCompleted += new RunWorkerCompletedEventHandler(worker_RunWorkerCompleted);         workers.Add(worker);     }       foreach(BackgroundWorker worker in workers)     {         worker.RunWorkerAsync();     } 

Can anyone enlighten me as to what is going on?

Thanks

EDIT: If as suggested I sleep for 5 seconds, instead of httpwebrequest, then I do get more threads firing but not as many as I would have expected:

01/09/09 11:56:14 Starting new HTTP request on 7 01/09/09 11:56:14 Starting new HTTP request on 11 01/09/09 11:56:15 Starting new HTTP request on 12 01/09/09 11:56:15 Starting new HTTP request on 13 01/09/09 11:56:16 Starting new HTTP request on 14 01/09/09 11:56:16 Starting new HTTP request on 15 01/09/09 11:56:17 Starting new HTTP request on 16 01/09/09 11:56:17 Starting new HTTP request on 17 01/09/09 11:56:18 Starting new HTTP request on 18 01/09/09 11:56:19 Starting new HTTP request on 7 01/09/09 11:56:19 7 has finished! 01/09/09 11:56:19 Starting new HTTP request on 11 01/09/09 11:56:19 11 has finished! 01/09/09 11:56:19 Starting new HTTP request on 19 01/09/09 11:56:20 Starting new HTTP request on 20 01/09/09 11:56:20 Starting new HTTP request on 12 01/09/09 11:56:20 12 has finished! 

It still looks like I'm only getting 2 threads starting every second, which seems mighty slow to me. I suppose the Console.WriteLine could be a problem?

EDIT: I set

ThreadPool.SetMinThreads(100, 4);  

and

System.Net.ServicePointManager.DefaultConnectionLimit = 100; 

and got the following results:

01/09/09 14:00:07 Starting new HTTP request on 11 01/09/09 14:00:07 Starting new HTTP request on 81 01/09/09 14:00:07 Starting new HTTP request on 82 01/09/09 14:00:07 Starting new HTTP request on 79 01/09/09 14:00:07 Starting new HTTP request on 83 01/09/09 14:00:07 Starting new HTTP request on 84 01/09/09 14:00:07 Starting new HTTP request on 85 01/09/09 14:00:07 Starting new HTTP request on 87 01/09/09 14:00:07 Starting new HTTP request on 88 ... 01/09/09 14:00:07 84 has finished! Took 323.0323 milliseconds 01/09/09 14:00:08 88 has finished! Took 808.0808 milliseconds 01/09/09 14:00:08 96 has finished! Took 806.0806 milliseconds 01/09/09 14:00:08 94 has finished! Took 806.0806 milliseconds 01/09/09 14:00:08 98 has finished! Took 801.0801 milliseconds 01/09/09 14:00:08 80 has finished! Took 799.0799 milliseconds 01/09/09 14:00:08 86 has finished! Took 799.0799 milliseconds 01/09/09 14:00:08 92 has finished! Took 799.0799 milliseconds 01/09/09 14:00:08 100 has finished! Took 812.0812 milliseconds 01/09/09 14:00:08 82 has finished! Took 1010.101 milliseconds 

so was able to push out a whole lot of web requests concurrently. Which seemed to queue (calling out to an STA COM+ server) so that's what I expected.

Thanks for your help

Answer 1

Are all (or most) of your requests going to the same host by any chance? There's a built-in limit on a per-host basis. You can change this in app.config in the system.Net connectionManagement element.

The other thing is that the thread pool only ramps up its number of threads gradually - it starts a new thread every half second, IIRC. Could that be what you're seeing? Try getting rid of HttpWebRequest from the equation - just sleep for a couple of seconds instead...

I suspect the latter problem is the one you're initially running into, but the first one is going to cause you problems as well.

Answer 2

There is a limit in the number of simultaneous outgoing HTTP connections. I think you can control this by using the System.Net.ServicePointManager.DefaultConnectionLimit static property before creating the HttpWebRequest objects.