How to set HttpWebRequest.Timeout for a large HTTP request in C#

Question

I'm not getting how to deal with HttpWebRequest.Timeout. Before, I used to set timeouts for Socket objects where it was straight-forward: Timeout set the maximum amount of time for sending or receiving a chunk of data. However, it seems HttpWebRequest.Timeout sets the timeout for the entire HTTP request. If the request is big (for instance, I'm uploading a large file with HTTP PUT), it may take hours. This leads me to setting:

...
request.Timeout = System.Threading.Timeout.Infinite;
Stream requestStream = request.GetRequestStream();
for (something)
{
  ...
  requestStream.Write(b, 0, b.Length);
}

However, doesn't this mean that if the network connection with the server gets stuck, I'll end up with requestStream.Write never throwing 'Operation timed out' exception? So that the concept of timeouts won't work in this case?

Ideally, I would want that .Timeout setting only affected a single requestStream.Write. I could have as many Write()'s as needed provided that each one never takes more than .Timeout value.

Or do I need to implement my own Socket-based mechanism to achieve that?

Also, when setting a breakpoint, I found that requestStream is actually ConnectStream instance having .Timeout=300000 (300 seconds). Doesn't this mean that Infinite is not actually THAT infinite and is limited to 300 seconds? For a large file and slow connection, it's fairly tough limitation.

Answer 1

There are two timeouts that plague us in processing a large file upload. HttpWebRequest.Timeout and HttpWebRequest.ReadWriteTimeout. We'll need to address both.

HttpWebRequest.ReadWriteTimeout

First, let's address HttpWebRequest.ReadWriteTimeout. We need to disable "write stream buffering".

httpRequest.AllowWriteStreamBuffering = false;

With this setting changed, your HttpWebRequest.ReadWriteTimeout values will magically work as you would like them to, you can leave them at the default value (5 minutes) or even decrease them. I use 60 seconds.

This problem comes about because, when uploading a large file, if the data is buffered by the .NET framework, your code will think the upload is finished when it's not, and call HttpWebRequest.GetResponse() too early, which will time out.

HttpWebRequest.Timeout

Now, let's address HttpWebRequest.Timeout.

Our second problem comes about because HttpWebRequest.Timeout applies to the entire upload. The upload process actually consists of three steps (here's a great article that was my primary reference):

1. A request to start the upload.
2. Writing of the bytes.
3. A request to finish the upload and get any response from the server.

If we have one timeout that applies to the whole upload, we need a large number to accomodate uploading large files, but we also face the problem that a legitimate timeout will take a long time to actually time out. This is not a good situation. Instead, we want a short time out (say 30 seconds) to apply to steps #1 and #3. We don't want an overall timeout on #2 at all, but we do want the upload to fail if bytes stop getting written for a period of time. Thankfully, we have already addressed #2 with HttpWebRequest.ReadWriteTimeout, we just need to fix the annoying behaviour of HttpWebRequest.Timeout. It turns out that the async versions of GetRequestStream and GetResponse do exactly what we need.

So you want this code:

public static class AsyncExtensions
{
    public static Task<T> WithTimeout<T>(this Task<T> task, TimeSpan timeout)
    {
        return Task.Factory.StartNew(() =>
        {
            var b = task.Wait((int)timeout.TotalMilliseconds);
            if (b) return task.Result;
            throw new WebException("The operation has timed out", WebExceptionStatus.Timeout);
        });
    }
}

And instead of calling GetRequestStream and GetResponse we'll call the async versions:

var uploadStream = httpRequest.GetRequestStreamAsync().WithTimeout(TimeSpan.FromSeconds(30)).Result;

And similarly for the response:

var response = (HttpWebResponse)httpRequest.GetResponseAsync().WithTimeout(TimeSpan.FromSeconds(30)).Result;

That's all you'll need. Now your uploads will be far more reliable. You might wrap the whole upload in a retry loop for added certainty.

Answer 2

Note that for HttpWebRequest with mono, this is very important (as stated above by Boinst) to use

httpRequest.AllowWriteStreamBuffering = false;

I spent an entire day with Wireshark chasing down why large file uploads were not successful, and finally stumbled on this quick, easy answer.