Connection was closed error between HttpClient and ASP.NET Core 2.0 webservice

Question

I have a ASP.NET Core 2.0 webservice running on IIS. One of controller's methods looks more or less like this:

[HttpGet()]
public IActionResult Test()
{
    // do some db updates and get data
    var result = DoSomeStuff();
    // serialize data to byte array
    var output = Serialize(result);

    return File(output, "application/octet-stream");
}

It does some database updates, query records from a table, serialize data and send them as a response. Data are sent in binary format. I'm using MessagePack-CSharp as a serializer.

Then I have client application that communicates with this webservice. It's .NET Standard 2.0 library, which is referenced from .NET 4.6.1 console app. I use HttpClient for requesting and HttpResponseMessage.Content.ReadAsByteArrayAsync() to read the response (exact code see below).

I wanted to do some tests. My table has cca. 80 columns and contains cca. 140000 records. All of them are supposed to be sent to client. Getting data from db takes few seconds, then it is everything serialized and result of cca. 34MB is sent to client.

I have 10 clients. When they call webservice serially, everything works. When I stress the webservice and fire clients in parallel, I almost always get an error on some of them (usually one or two fails, sometimes even 4-5).

Exception is following and it is raised from ReadAsByteArrayAsync call:

System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.ConnectStream.EndRead(IAsyncResult asyncResult)
   at System.IO.Stream.<>c.<BeginEndReadAsync>b__43_1(Stream stream, IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
...
---> (Inner Exception #0) System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)
   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at System.Net.ConnectStream.EndRead(IAsyncResult asyncResult)
   at System.IO.Stream.<>c.<BeginEndReadAsync>b__43_1(Stream stream, IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncTrimPromise`1.Complete(TInstance thisRef, Func`3 endMethod, IAsyncResult asyncResult, Boolean requiresSynchronization)
...

I found several SO threads related to such exception (e.g. here), so I initially thought this is a client related problem. Answers suggested:

• switching to HTTP 1.0
• setting Connection: close instead of Connection: keep-alive
• vice versa point above

Nothing worked for me. I think I read somewhere that there was some bug in HttpClient (cannot find the source now). I tried to use newest System.Net.Http package from Nuget. Same problem. I created .NET Core console app and use Core version of HttpClient. Same problem. I used HttpWebRequest instead of HttpClient. Same underlying problem.

I was running webservice and clients on the same VM machine. Just to rule out some local problems, I run clients simultaneously from other computers. Same problem.

So I ended up with following simplified code (just one app with 10 threads):

private async void Test_Click(object sender, RoutedEventArgs e)
{
    try
    {
        var tasks = Enumerable.Range(1, 10).Select(async i => await Task.Run(async () => await GetContent(i))).ToList();

        await Task.WhenAll(tasks);

        MessageBox.Show(String.Join(Environment.NewLine, tasks.Select(t => t.Result.ToString())));
    }
    catch (Exception ex)
    {
        MessageBox.Show(ex.ToString());
    }
}

private async Task<Int32> GetContent(Int32 id)
{
    using (var httpClient = new HttpClient())
    {
        var url = "http://localhost/TestService/api/test";

        using (var responseMessage = await httpClient.GetAsync(url).ConfigureAwait(false))
        {
            // just read everything and return length
            // ReadAsByteArrayAsync throws sometimes an exception
            var content = await responseMessage.Content.ReadAsByteArrayAsync();
            return content.Length;
        }
    }
}

I was curious about the actual traffic, so I setup Fiddler. When the error occurs, Fiddler shows that response is indeed corrupted and only part of supposed data amount has actually been sent (6MB, 20MB, ... instead of 34MB). Seems like it is interrupted randomly. I played a while with Wireshark and I saw that RST/ACK packet is sent from server, but I'm not good enough with analyzing such low level communication.

So, I focused on server side. Of course I double checked if there is any exception in controller's method. Everything works fine. I set the log level to trace and found following in log:

info: Microsoft.AspNetCore.Server.Kestrel[28]
      Connection id "0HL89D9NUNEOQ", Request id "0HL89D9NUNEOQ:00000001": the connection was closed becuase the response was not read by the client at the specified minimum data rate.
dbug: Microsoft.AspNetCore.Server.Kestrel[10]
      Connection id "0HL89D9NUNEOQ" disconnecting.
...
info: Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv[14]
      Connection id "0HL89D9NUNEOQ" communication error.
Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv.Internal.Networking.UvException: Error -4081 ECANCELED operation canceled

I didn't find anything interesting and ASP.NET Core specific related to this error. According to this documentation, IIS has an option to specify minimum throughput rate, when it sends a response to the client, with following setting:

<system.applicationHost>
  <webLimits minBytesPerSecond="0"/>
</system.applicationHost>

I use it in my Web.config, but it has no effect (is it applied to ASP.NET Core apps or is it full framework only setting?).

I tried to return FileStreamResult instead of FileContentResult, but again - it didn't help.

Similarly to client, I tried to find minimal reproducible code for server side too. Method just had Thread.Sleep(8000) (instead of db call), then generated random 50Mb byte array and returned it. This worked without any problem, so I guess I'll keep investigating in that direction. I'm aware that db might be bottleneck here, but not sure how it could cause this (no timeout exception, no deadlock, ...).

Any advises? I would at least like to know whether it is server or really client related problem.

Answer 1

It looks like your throughput drops below the minimum data rate. This behavior is described in Kestrel Fundamentals:

Kestrel checks every second if data is coming in at the specified rate in bytes/second. If the rate drops below the minimum, the connection is timed out. The grace period is the amount of time that Kestrel gives the client to increase its send rate up to the minimum; the rate is not checked during that time. The grace period helps avoid dropping connections that are initially sending data at a slow rate due to TCP slow-start.

The default minimum rate is 240 bytes/second, with a 5 second grace period.

A minimum rate also applies to the response. The code to set the request limit and the response limit is the same except for having RequestBody or Response in the property and interface names.

You can configure this in Program.cs like this:

var host = new WebHostBuilder() 
    .UseKestrel(options => 
    { 
        options.Limits.MinResponseDataRate = null;
    })

Setting this option to null indicates no minimum data rate should be enforced.