Multiple HTTP request in C#

Question

I need to send about 200 HTTP requests in parallel to different servers and get response. I use HttpWebRequest class in C#. But I don't see good time enhancement when requests are handled in parallel. For example if one request needs 3sec to get response, 2 request in parallel - 6sec, 3 requests - 8 secs, 4 requests - 11sec ... It is not good, I hope that there should be best time, about 10 sec for 200 requests. It looks like only 2-3 requests performs in parallel, but timeout starts immediately after WebRequest object creation. I tried set DefaultConnectionLimit and MaxServicePoints values, but id didn't help. As I understand these parameters for number of requests to one site in parallel. I need requests to different sites.

Code example that I use to test:

ArrayList a = new ArrayList(150);

for (i = 50; i < 250; i++ )
{
   a.Add("http://207.242.7." + i.ToString() + "/");
}

for (i = 0; i < a.Count; i++)
{
    Thread t = new Thread(new ParameterizedThreadStart(performRequest));
    t.Start(a[i]);
}


static void performRequest(object ip)
{
      HttpWebRequest req = (HttpWebRequest)WebRequest.Create((stirng)ip);

      HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
}

Сan anyone ever encountered such a problem? Thank you for any suggestions.

Answer 1

This might help - http://social.msdn.microsoft.com/forums/en-US/netfxnetcom/thread/1f863f20-09f9-49a5-8eee-17a89b591007

Suggests there is a limit on the number of TCP connections allowed, but that you can increase the limit

Answer 2

Instead of starting up your own threads try using the asynchronous methods of HttpWebRequest such as HttpWebRequest.BeginGetResponse and HttpWebRequest.BeginGetRequestStream.