Polly policy not working using "AddPolicyHandler"

Question

I have an application that makes a request for an authenticated service, where it is necessary to pass the access_token.

My idea is to use Polly to retry if the access_token is expired.

I'm using Refit (v5.1.67) and Polly (v7.2.1) in a .NET Core 3.1 application.

The services are registered as follows:

services.AddTransient<ExampleDelegatingHandler>();

IAsyncPolicy<HttpResponseMessage> retryPolicy = Policy<HttpResponseMessage>
    .Handle<ApiException>()
    .RetryAsync(1, (response, retryCount) =>
    {
        System.Diagnostics.Debug.WriteLine($"Polly Retry => Count: {retryCount}");
    });

services.AddRefitClient<TwitterApi>()
    .ConfigureHttpClient(c =>
    {
        c.BaseAddress = new Uri("https://api.twitter.com/");
    })
    .AddHttpMessageHandler<ExampleDelegatingHandler>()
    .AddPolicyHandler((sp, req) =>
    {
        //this policy does not works, because the exception is not catched on 
        //"Microsoft.Extensions.Http.PolicyHttpMessageHandler" (DelegatingHandler)
        return retryPolicy;
    });

public interface TwitterApi
{
    [Get("/2/users")]
    Task<string> GetUsers();
}

public class ExampleDelegatingHandler : DelegatingHandler
{
    protected async override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        try
        {
            return await base.SendAsync(request, cancellationToken);
        }
        catch (Exception)
        {
            //Why do not catch the exception?
            throw;
        }
    }
}

The retry policy is not working!

Analyzing the problem, I realized that the exception is not being caught inside the HttpClient's DelegatingHandler. Since the AddPolicyHandler statement is generating a DelegatingHandler (PolicyHttpMessageHandler) to execute the policy and the exception is not caught there, the policy never executes. I realized that the problem only occurs in asynchronous scenarios, where the request can be sent. In synchronous scenarios it works (example: timeout).

Why the exception is not caught inside DelegatingHandler??

I am attaching an example project simulating a Twitter call.

https://www.dropbox.com/s/q1797rq1pbjvcls/ConsoleApp2.zip?dl=0

External references:

https://github.com/reactiveui/refit#using-httpclientfactory

https://www.hanselman.com/blog/UsingASPNETCore21sHttpClientFactoryWithRefitsRESTLibrary.aspx

https://learn.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-3.1

Answer 1

I had an issue involving .NET 5 >= with Polly and HttpClient, which the compiler showed: HttpClientBuilder does not contain a definition for AddPolicyHandler. I could fix it when I changed the Nuget PackagePolly.Extensions.Http to Microsoft.Extensions.Http.Polly, I know that isn't the same situation reported here but it might be useful for other people who have come here to find this answer, like me.

Answer 2

TL;DR: The ordering of AddPolicyHandler and AddHttpMessageHandler does matter.

---

I've recreated the problem with HttpClient (so without Refit).

Typed HttpClient for testing

public interface ITestClient
{
    Task<string> Get();
}

public class TestClient: ITestClient
{
    private readonly HttpClient client;
    public TestClient(HttpClient client)
    {
        this.client = client;
    }
    public async Task<string> Get()
    {
        var resp = await client.GetAsync("http://not-existing.site");
        return "Finished";
    }
}

Controller for testing

[ApiController]
[Route("[controller]")]
public class TestController : ControllerBase
{
    private readonly ITestClient client;

    public TestController(ITestClient client)
    {
        this.client = client;
    }

    [HttpGet]
    public async Task<string> Get()
    {
        return await client.Get();
    }
}

DelegateHandler for testing

public class TestHandler: DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        try
        {
            return await base.SendAsync(request, cancellationToken);
        }
        catch (System.Exception ex)
        {
            _ = ex;
            throw;
        }
    }
}

---

Ordering #1 - Handler, Policy

Startup

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();
    services.AddTransient<TestHandler>();
    services.AddHttpClient<ITestClient, TestClient>()
        .AddHttpMessageHandler<TestHandler>() //Handler first
        .AddPolicyHandler(RetryPolicy()); //Policy second
}

private IAsyncPolicy<HttpResponseMessage> RetryPolicy()
    => Policy<HttpResponseMessage>
    .Handle<HttpRequestException>()
    .RetryAsync(1, (resp, count) =>
    {
        Console.WriteLine(resp.Exception);
    });

Execution order

1. TestController's Get
2. TestClient's Get
3. TestHandler's SendAsync's try
4. RetryPolicy's onRetry
5. TestHandler's SendAsync's catch
6. TestController's Get fails with HttpRequestException (inner: SocketException)

So, here the retry policy does not fired.

Ordering #2 - Policy, Handler

Startup

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllers();
    services.AddTransient<TestHandler>();
    services.AddHttpClient<ITestClient, TestClient>()
        .AddPolicyHandler(RetryPolicy()) //Policy first
        .AddHttpMessageHandler<TestHandler>(); //Handler second
}

private IAsyncPolicy<HttpResponseMessage> RetryPolicy()
    => Policy<HttpResponseMessage>
    .Handle<HttpRequestException>()
    .RetryAsync(1, (resp, count) =>
    {
        Console.WriteLine(resp.Exception);
    });

Execution order

1. TestController's Get
2. TestClient's Get
3. TestHandler's SendAsync's try
4. TestHandler's SendAsync's catch
5. RetryPolicy's onRetry
6. TestHandler's SendAsync's try
7. TestHandler's SendAsync's catch
8. TestController's Get fails with HttpRequestException (inner: SocketException)

So, here the retry policy has been fired.

Answer 3

1. Why

At the time policies and delegating handlers are executed, a failed HTTP response is not an exception yet. It's just an instance of HttpResponseMessage with an unsuccessful status. Refit converts this status into an exception as the very last step in the request-response processing.

2. Order

As correctly noted in Peter Csala's answer, order matters. When a request is made:

1. Refit serializes the parameters into an HttpRequestMessage and passes it to HttpClient
2. HttpClient does initial preparations
3. HttpClient runs the request messages through the handlers and policies in the order they were added to the client
4. The resulting message is sent to the server
5. Server's response is converted to a HttpResponseMessage object
6. This object bubbles up through the same sequence of handlers and policies but in reverse order
7. HttpClient does final processing and returns the result to Refit
8. Refit converts any errors into ApiExceptions

Therefore a retry policy will re-run everything that was added after it, but whatever was before it will be executed only once.

So if you want your access_token to be re-generated on every retry, the delegating handler that creates the token must be registered after the retry policy.

3. How

The easiest way to retry on HTTP failures is to use HttpPolicyExtensions.HandleTransientHttpError() from Polly.Extensions.Http. Otherwise you'd have to check for all the failure HTTP status codes yourself. The benefit of HandleTransientHttpError is that it only retries on failures which makes sense to retry, like 500 or socket errors. On the other hand it will not retry a 404 for instance, because the resource is not there and is unlikely to reappear if we try again.