.NET Core 3.1 HttpClient throws exception intermittently: SocketException: The I/O operation has been aborted because of either a thread exit or an

Question

Intermittently I get the exceptions below:

IOException: Unable to read data from the transport connection: The I/O operation has been aborted because of either a thread exit or an application request..

SocketException: The I/O operation has been aborted because of either a thread exit or an application request.

The system is querying an external resource and from time to time the exceptions happen without anything seeming to be out of the ordinary. I have tried to set a longer timeout for HttpClient but it did not help. It could be anywhere from 5000-50000 searches before the exception happens but I would still like to mitigate it. If I retry the same search directly after exception it works so the receiving party does not seem to have a problem even though I can't access that applications logs. Runs on .NET Core 3.1.

MyService.cs

public class MyService
{

    private readonly HttpClient _httpClient;

    public MyService(HttpClient client)
    {
        client.BaseAddress = new Uri("https://example.com/");
        client.Timeout = TimeSpan.FromMinutes(5);
        _httpClient = client;
    }
    
    private async Task<List<string>> GetValuesFromSearch(string search)
    {
        //Exception is thrown here
        var response = await _httpClient.GetAsync("search/" + search);

        using var responseStream = await response.Content.ReadAsStreamAsync();
        
        response.EnsureSuccessStatusCode();

        var searchResultList = await JsonSerializer.DeserializeAsync
            <List<string>>(responseStream);

        return searchResultList;
    }
}

Called like this:

var myService = new MyService(new HttpClient());

foreach (var search in listToIterate)
{
    //Can be called up to 200 000 times
    var result = await myService.GetValuesFromSearch(search);
}

Answer 1

The issue could be due to socket exhaustion. This is a known issue with HttpClient and the solution is to use HttpClientFactory. I haven't tested this but here's a quick re-write of your MyService class:

public class MyService
{
    private readonly IHttpClientFactory _httpClientFactory;

    public MyService(IHttpClientFactory httpClientFactory)
    {
        _httpClientFactory = httpClientFactory ??
            throw new ArgumentNullException(nameof(httpClientFactory));
    }

    private async Task<List<string>> GetValuesFromSearch(string search)
    {
        var _httpClient = _httpClientFactory.CreateClient("MyClient");

        _httpClient.BaseAddress = new Uri("https://example.com/");
        _httpClient.Timeout = TimeSpan.FromMinutes(5);

        // You could also set the above in Startup.cs or wherever you add your services:
        //services.AddHttpClient("MyClient", c => {
        //    c.BaseAddress = new Uri("https://example.com/");
        //    c.Timeout = TimeSpan.FromMinutes(5);
        //});

        //Exception is thrown here
        var response = await _httpClient.GetAsync("search/" + search);

        using var responseStream = await response.Content.ReadAsStreamAsync();

        response.EnsureSuccessStatusCode();

        var searchResultList = await JsonSerializer.DeserializeAsync
            <List<string>>(responseStream);

        return searchResultList;
    }
}

Answer 2

If your request fails to return a HttpResponseMessage, HttpClient will throw this as the inner exception stack trace of type TaskCancelledException. To confirm, try using Polly to add a TimeoutAsync policy; the exception should change to a TimeOutRejectedException.

In a similar use case, the best solution I have found is to:

• use HttpClientFactory, and skip the using statement when you use your client.
• use Polly to build in some additional resiliency.
• wrap the HttpClient code in a try/catch block so your process doesn't bomb out in the event there is no HttpResponseMessage to work with (this is the actual 'fix,' the other two pieces are to keep the exception from throwing as frequently):

private async Task<List<string>> GetValuesFromSearch(string search)
    {
        try
            {
                //Exception is thrown here
                var response = await _httpClient.GetAsync("search/" + search);

                var responseStream = await response.Content.ReadAsStreamAsync();

                response.EnsureSuccessStatusCode();

                var searchResultList = await JsonSerializer.DeserializeAsync
                    <List<string>>(responseStream);

                return searchResultList;
            }
        catch (Exception ex)
            {
                // Log the exception. 
                // Do what you want, or return null and handle a dropped request in your calling method.
            }
    }