System.Net.HttpClient: SendAsync failed with OperationCanceledException without http request throught network

Question

We use System.Net.Http.HttpClientfor calls between microservices inside k8s.

• OS Linux(inside docker)
• dotnet TargetFramework: netcoreapp2.1
• server: Kestrel
• protocol: http

Few days ago we noticed very strange behaviour of http calls: some calls between microservice(near 2-3%) failed with error

System.OperationCanceledException: The operation was canceled.
   at System.Net.Http.HttpClient.HandleFinishSendAsyncError(Exception e, CancellationTokenSource cts)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at OurCode.HttpServiceClient.GetStringResponseAsync(String methodUri, HttpMethod httpMethod)
   ...another our code...

after our timeout for http calls(it is 3 sec). But there was no logs about call inside callee service.

We enabled packetbeat for tracing http requests and also noticed, that no any http requests from caller service to callee service was executed. CPU, memory and network for this services was OK all the time.

Simplified version of our code for http calls looks like:

public async Task<string> GetStringResponseAsync(String methodUri,  HttpMethod httpMethod)
{
    int timeoutInMilliseconds = 3000;
    var tokenSource = new CancellationTokenSource();
    var rm = new HttpRequestMessage(httpMethod, methodUri);
    Task<HttpResponseMessage> httpTask = HttpClient.SendAsync(rm, tokenSource.Token);
    tokenSource.CancelAfter(timeoutInMilliseconds);
    HttpResponseMessage response = await httpTask;
    await EnsureSuccessStatusCode(response);
    return await response.Content.ReadAsStringAsync();
}

Any ideas about what problem can cause this strange behaviour without http request through network, and what can I do for further investigation?

Answer 1

It just meant that the web service did not respond.

HttpClient throws a TaskCanceledException, (which inherits from OperationCanceledException) when the timeout elapses. It is not intuitive and doesn't make any sense to me (and others), but that's what it does unfortunately.

There is some discussion about it here (a few people mentioned some workarounds to distinguish timeout from a true cancel, if you care).