Retrying HttpClient Unsuccessful Requests

Question

I am building a function that given an HttpContent Object, will issues request and retry on failure. However I get exceptions saying that HttpContent Object is disposed after issuing the request. Is there anyway to copy or duplicate the HttpContent Object so that I can issue multiple requests.

 public HttpResponseMessage ExecuteWithRetry(string url, HttpContent content)
 {
  HttpResponseMessage result = null;
  bool success = false;
  do
  {
      using (var client = new HttpClient())
      {
          result = client.PostAsync(url, content).Result;
          success = result.IsSuccessStatusCode;
      }
  }
  while (!success);

 return result;
} 

// Works with no exception if first request is successful
ExecuteWithRetry("http://www.requestb.in/xfxcva" /*valid url*/, new StringContent("Hello World"));
// Throws if request has to be retried ...
ExecuteWithRetry("http://www.requestb.in/badurl" /*invalid url*/, new StringContent("Hello World"));

(Obviously I don't try indefinitely but the code above is essentially what i want).

It yields this exception

System.AggregateException: One or more errors occurred. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Http.StringContent'.
   at System.Net.Http.HttpContent.CheckDisposed()
   at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context)
   at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Submission#8.ExecuteWithRetry(String url, HttpContent content)

Is there anyway to duplicate an HttpContent Object or reuse it?

Answer 1

Instead of implementing retry functionality that wraps the HttpClient, consider constructing the HttpClient with a HttpMessageHandler that performs the retry logic internally. For example:

public class RetryHandler : DelegatingHandler
{
    // Strongly consider limiting the number of retries - "retry forever" is
    // probably not the most user friendly way you could respond to "the
    // network cable got pulled out."
    private const int MaxRetries = 3;

    public RetryHandler(HttpMessageHandler innerHandler)
        : base(innerHandler)
    { }

    protected override async Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken)
    {
        HttpResponseMessage response = null;
        for (int i = 0; i < MaxRetries; i++)
        {
            response = await base.SendAsync(request, cancellationToken);
            if (response.IsSuccessStatusCode) {
                return response;
            }
        }

        return response;
    }
}

public class BusinessLogic
{
    public void FetchSomeThingsSynchronously()
    {
        // ...

        // Consider abstracting this construction to a factory or IoC container
        using (var client = new HttpClient(new RetryHandler(new HttpClientHandler())))
        {
            myResult = client.PostAsync(yourUri, yourHttpContent).Result;
        }

        // ...
    }
}

Answer 2

ASP.NET Core 2.1 Answer

ASP.NET Core 2.1 added support for Polly directly. Here UnreliableEndpointCallerService is a class which accepts a HttpClient in its constructor. Failed requests will retry with an exponential back-off so that the next retry takes place in an exponentially longer time after the previous one:

services
    .AddHttpClient<UnreliableEndpointCallerService>()
    .AddTransientHttpErrorPolicy(
        x => x.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)));

Also, consider reading my blog post "Optimally Configuring HttpClientFactory".

Other Platforms Answer

This implementation uses Polly to retry with an exponential back-off so that the next retry takes place in an exponentially longer time after the previous one. It also retries if a HttpRequestException or TaskCanceledException is thrown due to a timeout. Polly is much easier to use than Topaz.

public class HttpRetryMessageHandler : DelegatingHandler
{
    public HttpRetryMessageHandler(HttpClientHandler handler) : base(handler) {}

    protected override Task<HttpResponseMessage> SendAsync(
        HttpRequestMessage request,
        CancellationToken cancellationToken) =>
        Policy
            .Handle<HttpRequestException>()
            .Or<TaskCanceledException>()
            .OrResult<HttpResponseMessage>(x => !x.IsSuccessStatusCode)
            .WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)))
            .ExecuteAsync(() => base.SendAsync(request, cancellationToken));
}

using (var client = new HttpClient(new HttpRetryMessageHandler(new HttpClientHandler())))
{
    var result = await client.GetAsync("http://example.com");
}

Answer 3

The current answers won't work as expected in all cases, specifically in the very common case of request timeout (see my comments there).

In addition, they implement a very naive retry strategy - many times you'd want something a bit more sophosticated, such as exponential backoff (which is the default in the Azure Storage Client API).

I stumbled upon TOPAZ while reading a related blog post (also offering the misguided internal retry approach). Here's what I came up with:

// sample usage: var response = await RequestAsync(() => httpClient.GetAsync(url));
Task<HttpResponseMessage> RequestAsync(Func<Task<HttpResponseMessage>> requester)
{
    var retryPolicy = new RetryPolicy(transientErrorDetectionStrategy, retryStrategy);
    //you can subscribe to the RetryPolicy.Retrying event here to be notified 
    //of retry attempts (e.g. for logging purposes)
    return retryPolicy.ExecuteAsync(async () =>
    {
        HttpResponseMessage response;
        try
        {
            response = await requester().ConfigureAwait(false);
        }
        catch (TaskCanceledException e) //HttpClient throws this on timeout
        {
            //we need to convert it to a different exception
            //otherwise ExecuteAsync will think we requested cancellation
            throw new HttpRequestException("Request timed out", e);
        }
        //assuming you treat an unsuccessful status code as an error
        //otherwise just return the respone here
        return response.EnsureSuccessStatusCode(); 
    });
}

Note the requester delegate parameter. It should not be an HttpRequestMessage since you can't send the same request multiple times. As for the strategies, that depends on your use case. For example, a transient error detection strategy could be as simple as:

private sealed class TransientErrorCatchAllStrategy : ITransientErrorDetectionStrategy
{
    public bool IsTransient(Exception ex)
    {
        return true;
    }
}

As for the retry strategy, TOPAZ offers three options:

1. FixedInterval
2. Incremental
3. ExponentialBackoff

For example, here's the TOPAZ equivalent of what the Azure Client Storage Library uses for default:

int retries = 3;
var minBackoff = TimeSpan.FromSeconds(3.0);
var maxBackoff = TimeSpan.FromSeconds(120.0);
var deltaBackoff= TimeSpan.FromSeconds(4.0);
var strategy = new ExponentialBackoff(retries, minBackoff, maxBackoff, deltaBackoff);

For more information see http://msdn.microsoft.com/en-us/library/hh680901(v=pandp.50).aspx

EDIT Note that if your request contains an HttpContent object, you'll have to regenerate it every time as that will be disposed by HttpClient as well (thanks for catching that Alexandre Pepin). For example () => httpClient.PostAsync(url, new StringContent("foo"))).

Answer 4

Duplicating the StringContent isn't probably the best idea. But simple modification could fix the problem. Just modify the function and create the StringContent object inside of the loop, something like:

public HttpResponseMessage ExecuteWithRetry(string url, string contentString)
{
   HttpResponseMessage result = null;
   bool success = false;
   using (var client = new HttpClient())
   {
      do
      {
         result = client.PostAsync(url, new StringContent(contentString)).Result;
         success = result.IsSuccessStatusCode;
      }
      while (!success);
  }    

  return result;
} 

and then call it

ExecuteWithRetry("http://www.requestb.in/xfxcva" /*valid url*/, "Hello World");