Fastest way to download multiple webpages using C#

Question

This is a (basic) example of what I currently have:

foreach (var uri in uris)
{
    using (var client = new WebClient())
    {
        client.Proxy = null;
        client.DownloadStringCompleted += DownloadComplete;
        client.DownloadStringAsync(uri);
    }
}

Is there a faster way?

Answer 1

The important thing is to make the downloads in parallel, which you are already doing thanks to the Async download.

The download speed of your code is entirely dependent of the actual network transfer speed, so it is as good as it gets.

Answer 2

I believe you can make it a lot faster if you set Accept-Encoding header to gzip,deflate, if the server support gzip (modern web server should support).

The basic idea is to ask the server zip the content before downloading, normally for a common web page, you may get 50% less in size and hence you can save 50% time.

Look at this: http://csharpfeeds.com/post/5518/HttpWebRequest_and_GZip_Http_Responses.aspx