WebRequest fails to download large files (~ 1 GB) properly

Question

I am attempting to download a large file from a public URL. It seemed to work fine at first but 1 / 10 computers seem to timeout. My initial attempt was to use WebClient.DownloadFileAsync but because it would never complete I fell back to using WebRequest.Create and read the response streams directly.

My first version of using WebRequest.Create found the same problem as WebClient.DownloadFileAsync. The operation times out and the file does not complete.

My next version added retries if the download times out. Here is were it gets weird. The download does eventually finish with 1 retry to finish up the last 7092 bytes. So the file is downloaded with exactly the same size BUT the file is corrupt and differs from the source file. Now I would expect the corruption to be in the last 7092 bytes but this is not the case.

Using BeyondCompare I have found that there are 2 chunks of bytes missing from the corrupt file totalling up to the missing 7092 bytes! This missing bytes are at 1CA49FF0 and 1E31F380, way way before the download times out and is restarted.

What could possibly be going on here? Any hints on how to track down this problem further?

Here is the code in question.

public void DownloadFile(string sourceUri, string destinationPath)
{
    //roughly based on: http://stackoverflow.com/questions/2269607/how-to-programmatically-download-a-large-file-in-c-sharp
    //not using WebClient.DownloadFileAsync as it seems to stall out on large files rarely for unknown reasons.

    using (var fileStream = File.Open(destinationPath, FileMode.Create, FileAccess.Write, FileShare.Read))
    {
        long totalBytesToReceive = 0;
        long totalBytesReceived = 0;
        int attemptCount = 0;
        bool isFinished = false;

        while (!isFinished)
        {
            attemptCount += 1;

            if (attemptCount > 10)
            {
                throw new InvalidOperationException("Too many attempts to download. Aborting.");
            }

            try
            {
                var request = (HttpWebRequest)WebRequest.Create(sourceUri);

                request.Proxy = null;//http://stackoverflow.com/questions/754333/why-is-this-webrequest-code-slow/935728#935728
                _log.AddInformation("Request #{0}.", attemptCount);

                //continue downloading from last attempt.
                if (totalBytesReceived != 0)
                {
                    _log.AddInformation("Request resuming with range: {0} , {1}", totalBytesReceived, totalBytesToReceive);
                    request.AddRange(totalBytesReceived, totalBytesToReceive);
                }

                using (var response = request.GetResponse())
                {
                    _log.AddInformation("Received response. ContentLength={0} , ContentType={1}", response.ContentLength, response.ContentType);

                    if (totalBytesToReceive == 0)
                    {
                        totalBytesToReceive = response.ContentLength;
                    }

                    using (var responseStream = response.GetResponseStream())
                    {
                        _log.AddInformation("Beginning read of response stream.");
                        var buffer = new byte[4096];
                        int bytesRead = responseStream.Read(buffer, 0, buffer.Length);
                        while (bytesRead > 0)
                        {
                            fileStream.Write(buffer, 0, bytesRead);
                            totalBytesReceived += bytesRead;
                            bytesRead = responseStream.Read(buffer, 0, buffer.Length);
                        }

                        _log.AddInformation("Finished read of response stream.");
                    }
                }

                _log.AddInformation("Finished downloading file.");
                isFinished = true;
            }
            catch (Exception ex)
            {
                _log.AddInformation("Response raised exception ({0}). {1}", ex.GetType(), ex.Message);
            }
        }
    }
}

Here is the log output from the corrupt download:

Request #1.
Received response. ContentLength=939302925 , ContentType=application/zip
Beginning read of response stream.
Response raised exception (System.Net.WebException). The operation has timed out.
Request #2.
Request resuming with range: 939295833 , 939302925
Received response. ContentLength=7092 , ContentType=application/zip
Beginning read of response stream.
Finished read of response stream.
Finished downloading file.

Answer 1

this is the method I usually use, it hasn't failed me so far for the same kind of loading you need. Try using my code to change yours up a bit and see if that helps.

if (!Directory.Exists(localFolder))
{
    Directory.CreateDirectory(localFolder);   
}


try
{
    HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(Path.Combine(uri, filename));
    httpRequest.Method = "GET";

    // if the URI doesn't exist, exception gets thrown here...
    using (HttpWebResponse httpResponse = (HttpWebResponse)httpRequest.GetResponse())
    {
        using (Stream responseStream = httpResponse.GetResponseStream())
        {
            using (FileStream localFileStream = 
                new FileStream(Path.Combine(localFolder, filename), FileMode.Create))
            {
                var buffer = new byte[4096];
                long totalBytesRead = 0;
                int bytesRead;

                while ((bytesRead = responseStream.Read(buffer, 0, buffer.Length)) > 0)
                {
                    totalBytesRead += bytesRead;
                    localFileStream.Write(buffer, 0, bytesRead);
                }
            }
        }
    }
}
catch (Exception ex)
{        
    throw;
}