Receiving error 'The request message was already sent' when using Polly

Question

I am currently using Polly to limit the number of request I send. This is the policy I currently have:

private AsyncPolicyWrap<HttpResponseMessage> DefineAndRetrieveResiliencyStrategy()
{
    HttpStatusCode[] retryCodes = {
       HttpStatusCode.InternalServerError,
       HttpStatusCode.BadGateway,
       HttpStatusCode.GatewayTimeout
    };
    
    var waitAndRetryPolicy = Policy
        .HandleResult<HttpResponseMessage>(e => e.StatusCode == HttpStatusCode.ServiceUnavailable || e.StatusCode == (HttpStatusCode)429)
        .WaitAndRetryAsync(10,
            attempt => TimeSpan.FromSeconds(5), (exception, calculatedWaitDuration) =>
            {
                _log.Info($"Bitfinex API server is throttling our requests. Automatically delaying for {calculatedWaitDuration.TotalMilliseconds}ms");
            }
        );
    
    var circuitBreakerPolicyForRecoverable = Policy
        .Handle<HttpResponseException>()
        .OrResult<HttpResponseMessage>(r => retryCodes.Contains(r.StatusCode))
        .CircuitBreakerAsync(
            handledEventsAllowedBeforeBreaking: 3,
            durationOfBreak: TimeSpan.FromSeconds(3),
            onBreak: (outcome, breakDelay) =>
            {
                _log.Info($"Polly Circuit Breaker logging: Breaking the circuit for {breakDelay.TotalMilliseconds}ms due to: {outcome.Exception?.Message ?? outcome.Result.StatusCode.ToString()}");
                
            },
            onReset: () => _log.Info("Polly Circuit Breaker logging: Call ok... closed the circuit again"),
            onHalfOpen: () => _log.Info("Polly Circuit Breaker logging: Half-open: Next call is a trial")
        );
    
    return Policy.WrapAsync(waitAndRetryPolicy, circuitBreakerPolicyForRecoverable);
}

I have thee following request sender:

private async Task<string> SendRequest(GenericRequest request, string httpMethod, string publicKey, string privateKey)
{
    var resiliencyStrategy = DefineAndRetrieveResiliencyStrategy();

    using (var client = new HttpClient())
    using (var httpRequest = new HttpRequestMessage(new HttpMethod(httpMethod), request.request))
    {
        string json = JsonConvert.SerializeObject(request);
        string json64 = Convert.ToBase64String(Encoding.UTF8.GetBytes(json));
        byte[] data = Encoding.UTF8.GetBytes(json64);

        client.BaseAddress = new Uri(Properties.Settings.Default.BitfinexUri);

        var hashMaker = new HMACSHA384(Encoding.UTF8.GetBytes(privateKey));
        byte[] hash = hashMaker.ComputeHash(data);
        string signature = GetHexString(hash);
        
        httpRequest.Headers.Add("X-BFX-APIKEY", publicKey);
        httpRequest.Headers.Add("X-BFX-PAYLOAD", json64);
        httpRequest.Headers.Add("X-BFX-SIGNATURE", signature);
        
        var message = await resiliencyStrategy.ExecuteAsync(() => client.SendAsync(httpRequest));
        var response = message.Content.ReadAsStringAsync().Result;

        return response;
    }
}

As soon as the code hits the waitAndRetryPolicy and awaits the required amount of time, I get the following error:

System.InvalidOperationException: 'The request message was already sent. Cannot send the same request message multiple times.'

I understand that this is happening because I am sending the same HttpRequest again but shouldn't the Polly library handle such an issue?

Answer 1

That exception:

System.InvalidOperationException: 'The request message was already sent. Cannot send the same request message multiple times.'

is thrown by the internals of HttpClient if you call directly into any .SendAsync(...) overload with an HttpRequestMessage which has already been sent.

If you are using .NET Core, the recommended solution is to use Polly with HttpClientFactory: this solves the above exception by executing the policy (for example retry) via a DelegatingHandler within HttpClient. It also solves the socket-exhaustion problem which can be caused by a frequent create/dispose of HttpClient, which the code posted in the question may be vulnerable to.

If you using .NET framework, the recommended solutions are:

• replicate the way HttpClientFactory places the policy in a DelegatingHandler; or
• refactor your code to manufacture a new instance of HttpRequestMessage (or clone the existing instance) within the code executed through the policy.

This stackoverflow question discusses the problem extensively and many variants on the above solutions.