SendAsync and CopyToAsync not working when downloading a large file

Question

I have a small app that receives a request from a browser, copy the header received and the post data (or GET path) and send it to another endpoint.

It then waits for the result and sends it back to the browser. It works like a reverse proxy.

Everything works fine until it receives a request to download a large file. Something like a 30MB will cause an strange behaviour in the browser. When the browser reaches around 8MB it stops receiving data from my app and, after some time, it aborts the download. Everything else works just fine.

If I change the SendAsync line to use HttpCompletionOption.ResponseContentRead it works just fine. I am assuming there is something wrong waiting for the stream and/or task, but I can't figure out what is going on.

The application is written in C#, .net Core (latest version available).

Here is the code (partial)

private async Task SendHTTPResponse(HttpContext context, HttpResponseMessage responseMessage)
{
    context.Response.StatusCode = (int)responseMessage.StatusCode;

    foreach (var header in responseMessage.Headers)
    {
        context.Response.Headers[header.Key] = header.Value.ToArray();
    }

    foreach (var header in responseMessage.Content.Headers)
    {
        context.Response.Headers[header.Key] = header.Value.ToArray();
    }

    context.Response.Headers.Remove("transfer-encoding");

    using (var responseStream = await responseMessage.Content.ReadAsStreamAsync())
    {
       await responseStream.CopyToAsync(context.Response.Body);
    }

}

public async Task ForwardRequestAsync(string toHost, HttpContext context)
{

    var requestMessage = this.BuildHTTPRequestMessage(context);
    var responseMessage = await _httpClient.SendAsync(requestMessage, HttpCompletionOption.ResponseHeadersRead, context.RequestAborted);
    await this.SendHTTPResponse(context, responseMessage);
}

EDIT

Changed the SendHTTPResponse to wait for responseMessage.Content.ReadAsStreamAsync using await operator.

Answer 1

Just a guess but I believe the issue lies with the removal of the transfer encoding:

context.Response.Headers.Remove("transfer-encoding");

If the http request you are making with _httpClient returns the 30MB file using Chunked encoding (target server doesn't know the file size) then you would need to return the file to the browser with Chunked encoding as well.

When you buffer the response on your webservice (by passing HttpCompletionOption.ResponseContentRead) you know the exact message size you are sending back to the browser so the response works successfully.

I would check the response headers you get from responseMessage to see if the transfer encoding is chunked.