Recommendations for executing .NET HttpWebRequests in parallel in ASP.NET

Question

I have an ASP.NET MVC web application that makes REST style web service calls to other servers. I have a scenario where I am making two HttpWebRequest calls to two separate services. I need them both to complete to continue, but their order doesn't matter. They may take 1-2 seconds each and I am running them in sequence now. Running them in parallel would decrease the user response time, but what is the best way?

In researching this, I can think of several options:

• Execute one request on the main thread, and spin up a second thread for the other request. Should created new threads or use a thread pool? If I use a pool, how do I size it? Also, not sure how I can join the threads back together (e.g. use ThreadPool.RegisterWaitForSingleObject)?
• Try and leverage the built-in IAsyncResult support for one or both of the requests. Again, not sure on which threads the async request execute, so not sure how to size the thread pool. How do I join IAsyncResult back into my main thread? All the examples I find process information in the callback, but I can just wait in my main thread and use the IsCompleted property?

I need to find a solution that will both function, and perform at scale. That is why I am worried about thread pool sizing. I would hate to have requests blocking because they are waiting for available threads.

Answer 1

I like to do this kind of thing a little more manually, rather than relying on asynchronous web requests or the thread pool's automatic sizing (25 threads by default). Of course, those are perfectly fine ways to solve your problem, but I think the following code is a bit more readable (in the example below, _links would contain a list of your links before processing occurs...):

private static IList<String> _links = new List<String>();
private const int NumberOfThreads = 2;

public void SpawnWebRequests()
{
    IList<Thread> threadList = new List<Thread>();

    for (int i = 0; i < NumberOfThreads; i++)
    {
        var thread = new Thread(ProcessWebRequests);
        threadList.Add(thread);
        thread.Start();
    }

    for (int i = 0; i < NumberOfThreads; i++)
    {
        threadList[i].Join();
    }
}

private static void ProcessWebRequests()
{
    String link;

    while (true)
    {
        lock(_links)
        {
            if (_links.Count == 0)
                break;

            link = _links.RemoveAt(0);
        }

        ProcessWebRequest(link);
    }
}

private static void ProcessWebRequest(String link)
{
    try
    {
        var request = (HttpWebRequest)WebRequest.Create(link);
        request.Method = "HEAD"; // or "GET", since some sites (Amazon) don't allow HEAD
        request.Timeout = DefaultTimeoutSeconds * 1000;

        // Get the response (throws an exception if status != 200)
        using (var response = (HttpWebResponse)request.GetResponse())
        {
            if (response.StatusCode == HttpStatusCode.OK)
                Log.Debug("Working link: {0}", request.RequestUri);
        }
    }
    catch (WebException ex)
    {
        var response = ((HttpWebResponse)ex.Response);
        var status = response != null
                         ? response.StatusCode
                         : HttpStatusCode.RequestTimeout;

        Log.WarnException(String.Format("Broken link ({0}): {1}", status, link), ex);

        // Don't rethrow, as this is an expected exception in many cases
    }
    catch (Exception ex)
    {
        Log.ErrorException(String.Format("Error processing link {0}", link), ex);

        // Rethrow, something went wrong
        throw;
    }
}

If you just want to manage the size of the thread pool (if you are using ThreadPool.QueueUserWorkItem()), you can use ThreadPool.SetMaxThreads = 2).

Of course, if you want to use the Microsoft-sanctioned async approach, check out this example: http://msdn.microsoft.com/en-us/library/86wf6409.aspx. Just be sure you clean up each response (via a "using" block or by closing the response object)!

Hope that helps, Noah