HttpWebRequest - can I do multiple calls at the sametime from multiple threads

Question

I use HttpWebRequest to create requests for webpages, than to parse them.

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(address);

then if more threads call

HttpWebResponse response = (HttpWebResponse)request.GetResponse()

at the same time, should each get it's own response or is it possible for thread 2 to get the response for thread7 for instance?

Obs: the address is the same for all threads, only the POST parameters change

 public class CheckHelper
{
    public  string GetPOSTWebsiteResponse(string WebAddress, string year)
    {
        StringBuilder QuerryData = new StringBuilder();
        String ResponseString;
        QuerryData.Append("forYear"+ "=" + year);

        #region build request
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(WebAddress);
        // Set the Method property of the request to POST
        request.Method = "POST";

        NameValueCollection headers = request.Headers;
        Type t = headers.GetType();
        PropertyInfo p = t.GetProperty("IsReadOnly", BindingFlags.Instance | BindingFlags.IgnoreCase | BindingFlags.NonPublic | BindingFlags.FlattenHierarchy);
        p.SetValue(headers, false, null);
        byte[] byteArray = Encoding.UTF8.GetBytes(QuerryData.ToString());
        request.ContentType = "application/x-www-form-urlencoded";
        request.ContentLength = byteArray.Length;

        #endregion

        // Get the request stream.
        using (Stream requestStream = request.GetRequestStream())
        {
            // Write the data to the request stream.
            requestStream.Write(byteArray, 0, byteArray.Length);
            // Close the Stream object.
        }



        #region get response
        using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
        {
            //Get the stream containing content returned by the server.
            using (var responseStream = response.GetResponseStream())
            {
                // Open the stream using a StreamReader for easy access.
                using (StreamReader responseReader = new StreamReader(responseStream))
                {
                    // Read the content.
                    ResponseString = responseReader.ReadToEnd();


                }
            }
        }

        #endregion        
        return ResponseString;
      }
}

this is how I use the method:

            Dictionary<int, Thread> threads=new Dictionary<int,Thread>();
            foreach (var year in AvailableYears)
            {
                threads[year] = new Thread(delegate()
                    {
                    var client=new CheckHelper(); 
                    string response=client.GetPOSTWebsiteResponse("http://abc123.com", year.ToString())
                    //The thread for year 2003 may get the response for the year 2007

                    responsesDictionary[year]=response;
                    });
                threads[year].Start();

            }
            //this is to force the main thread to wait until all responses are    received:
        foreach(var th in threads.Values){
                th.Join(10000);
            }

Please tell me where I mistake? How should I change the code? Please help, I can't find anything helpful on the web!

Answer 1

To be honest, I don't believe the multi-threading you're attempting to do is going to get you any performance gains. And seeing that there is no threshold for the number of threads you could be creating here the potential for worse performance than a single threaded (sequential) operation exists.

The ideal situation is that you have an asynchronous work flow. where your loop is something like this:

GetAsyncRequest MakeAsyncRequest ReceiveResponseAsync ProcessResponse WaitForAllRequestProcessingToComplete(Optionally)

Such that the result of each step feeds into the next (if there is a result) and the next. And you are processing the responses as soon as you receive them rather than accumulating (joining/blocking) all responses before proceeding to process them. This kind of thing can be easily done with Tasks and ContinueWith in .NET 4.0 and seeing that you've using .NET 4.0 I'd strongly suggest you do it as described above.

But, if you're unable to convert your processing into an async workflow, then...

The method shown below is the method that makes the call out to the Url and returns a response. The method uses Async calls but is blocking since your design seems to be as such.

static string GetWebResponse(string url, NameValueCollection parameters)
{
  var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
  httpWebRequest.ContentType = "application/x-www-form-urlencoded";
  httpWebRequest.Method = "POST";

  var sb = new StringBuilder();
  foreach (var key in parameters.AllKeys)
    sb.Append(key + "=" + parameters[key] + "&");
  sb.Length = sb.Length - 1;

  byte[] requestBytes = Encoding.UTF8.GetBytes(sb.ToString());
  httpWebRequest.ContentLength = requestBytes.Length;

  using (var requestStream = httpWebRequest.GetRequestStream())
  {
    requestStream.Write(requestBytes, 0, requestBytes.Length);
    requestStream.Close();
  }

  Task<WebResponse> responseTask = Task.Factory.FromAsync<WebResponse>(httpWebRequest.BeginGetResponse, httpWebRequest.EndGetResponse, null);
  using (var responseStream = responseTask.Result.GetResponseStream())
  {
    var reader = new StreamReader(responseStream);
    return reader.ReadToEnd();
  }
}

You'd call it like this:

  ServicePointManager.DefaultConnectionLimit = 20;//Please test different numbers here
  var tasks = new List<Task<string>>();
  for (int i = 1990; i < 2090; i++)
  {
    var postParameters = new NameValueCollection();
    postParameters.Add("data", i.ToString());
    tasks.Add(Task.Factory.StartNew(() => { return GetWebResponse("http://www.abc123.com", postParameters); }));
  }
  Task.WaitAll(tasks.ToArray());
  //At this point tasks[0].Result will be the result (The Response) of the first task
  //tasks[1].Result will be the result of the second task and so on.

See if this works for you.

If you truly need multi-threaded capability, of course seeing that you're hitting only one site, one would have to measure the performance benefits since the site needs to be able to handle the onslaught of requests and on the client the cost of creating threads, only to do some I/O bound task could end up being too costly and end with no performance gains.

Besides, without adjusting the DefaultConnectionLimit in the ServicePointManager, you'll never get more than 2 threads either ways since you're going against one domain and the default limit is 2 threads per domain.

I'd stick with the code I've presented and if there is a performance problem only then would I look at doing it some other way.

Edit: when using Async I/O you're not using worker threads but I/O thread. So basically you don't want to use QueueUserWorkItem (to create threads) or you're not creating threads yourself.

The code I've presented is using Async I/O and if going to execute multiple requests simutaneously and as fast as possible.

The for loop (in the second code listing) will finish almost immediately even though in the example it loops for 100 iterations, and will then wait for all I/O requests to finish. The ThreadPool and OS will handle running the I/O jobs as soon as and as fast as possible. In fact because these jobs are I/O bound, you won't see your CPU utilization go up either (unless you're doing cpu bound work later on).

Just play with the ServiceManager.DefaultConnectionLimit to get more speedup if needed. Note that this affects the service (the server) as well, since if you make a lot of simultaneous requests the server you're calling gets heavily loaded and that may not be something you want. So it's a balance you need to strike.

After the call to Task.WaitAll, you can iterate over your tasks collection and get at the results of each task using the syntax shown in the commented line in the code listing.