.NET HttpClient - Accept partial response when response header has an incorrect Content-Length

Question

I am working on an ASP.NET web application with .NET Core 3.1. The application downloads mp3 files from an external webserver which has a bug: The Content-Length in the response header reports a byte count which is higher than the mp3's actual byte count.

Here's an example with using curl to download a file from that server:

curl -sSL -D - "http://example.com/test.mp3" -o /dev/null
HTTP/1.1 200 OK
Cache-Control: private
Pragma: no-cache
Content-Length: 50561024
Content-Type: audio/mpeg
Content-Range: bytes 0-50561023/50561024
Expires: 0
Accept-Ranges: 0-50561023
Server: Microsoft-IIS/10.0
Content-Transfer-Encoding: binary
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Fri, 03 Jan 2020 23:43:54 GMT

curl: (18) transfer closed with 266240 bytes remaining to read

So even though curl reports an incomplete transfer, the mp3 is fully downloaded with 50294784 bytes and I can open it in any audio player I tried.

What I want in my web application is the same behavior as with curl: Ignore the incorrect Content-Length and download the mp3 until the server closes the transfer.

Right now I'm simply using HttpClient for asynchronously downloading the mp3:

internal static HttpClient httpClient = new HttpClient() { Timeout = new TimeSpan( 0, 15, 0 ) };

using( var response = await httpClient.GetAsync( downloadableMp3.Uri, HttpCompletionOption.ResponseContentRead ) )
using( var streamToReadFrom = await response.Content.ReadAsStreamAsync() )

However, unlike curl the transfer is aborted as a whole when the transfer is closed too early:

Task <SchedulerTaskWrapper FAILED System.Net.Http.HttpRequestException: Error while copying content to a stream.
 ---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)

Is there any way I can configure HttpClient to "ignore" the incorrect Content-Length and get the mp3 anyway?

Answer 1

If you look at the method SendAsyncCore in dotnet runtime repo, you can see quite large code that implements core functionality of sending requests and handling responses. If the server sends the content-length header, this method internally creates ContentLengthReadStream. This stream expects a fixed number of bytes and is read until the expected amount is reached. If the content-length is greater than the real amount of bytes then ContentLengthReadStream throws an exception with the message The response ended prematurely.

As all of those methods are quite rigid and internal, there is no room to extend or change this functionality. But there is a workaround for that. You can read the stream manually into your buffer, until the exception is thrown. The normal termination condition for stream would be that Read method returns zero bytes. This condition should be also included if content-length is correct.

using var resp = await httpClient.GetAsync("http://example.com/test.mp3", HttpCompletionOption.ResponseHeadersRead);
using var contentStream = await resp.Content.ReadAsStreamAsync();

var bufferSize = 2048;
var buffer = new byte[bufferSize];
var result = new List<byte>();

try
{
    var readBytes = 0;
    while ((readBytes = contentStream.Read(buffer)) != 0)
    {
        for (int i = 0; i < readBytes; i++)
        {
            result.Add(buffer[i]);
        }
    }
}
catch (IOException ex)
{
    if (!ex.Message.StartsWith("The response ended prematurely"))
    {
        throw;
    }
}

the above code loads the entire response bytes into List result. It might be not a good solution for large contents.

Also note that you shouldn't use HttpCompletionOption.ResponseContentRead in this case, because if you call GetAsync method it tries to read the content immediately. As we want to read the content later, this should be changed to HttpCompletionOption.ResponseHeadersRead. This means that GetAsync completes the operation when headers are read (while the content is not read yet).