HttpClient not sending Accept-Encoding on Windows 2008 R2

Question

I have a .NET Core 2.0 console app that does a GET request.

It seems like the published version does not send Accept-Encoding headers for compression on a test machine, but works on my local machine.

I cannot find any other pre-requesites that would make compression fail. Both are running the .NET Core 2.1.4 SDK.

I have tested the console app by running dotnet Console.dll in both environments.

1. Publish in VS2017
2. Go to output folder and run dotnet Console.dll. Verify header present in Fiddler.
3. Copy entire output folder and deploy onto server
4. Run dotnet Console.dll again and verify header missing on the server with Fiddler.

I have tried both HttpClient and RestSharp and I'm pretty perplexed.

Proof of concept that goes to a page that echoes the request headers:

 var handler = new HttpClientHandler()
            {
                AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip
            };

 using (var client = new HttpClient(handler))
 {
      response = client.GetStringAsync("http://scooterlabs.com/echo").Result;
 }

Local environment (Win10)

GET http://scooterlabs.com/echo HTTP/1.1
Connection: Keep-Alive
Accept-Encoding: gzip, deflate
Host: scooterlabs.com

Server (Win2008 R2 on AWS)

GET http://scooterlabs.com/echo HTTP/1.1
Connection: Keep-Alive
Host: scooterlabs.com

Answer 1

My best guess would be that's because WinHttp library, which is used by default by HttpClient on windows does not support gzip\deflate on windows versions before Windows 8.1+. When supported - WinHttp will also set Accept-Encoding header. So on Windows Server 2008 when .NET routes request through WinHttp - it either sets this option and it is ignored, or it checks if this option is supported and if not - just not sets it.

If you set this header manually (like client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));) - option is still ignored but header passes through and server returns compressed response. If supported - WinHttp will decompress that response and remove Content-Encoding header, so response will arrive decompressed to .NET. If not supported - response will arrive compressed and .NET itself will decompress it if you set AutomaticDecompression.

So to conclude - on Windows versions before 8.1 it seems you need to set both AutomaticDecompression and related Accept-Encoding headers for this to work as expected.

Answer 2

Building on Ivan and Evk's answers, it does appear that this is an issue specific to older versions of Windows (older than Win8.1). Here is how to work around and successfully handle compression on older versions of Windows.

   var handler = new HttpClientHandler()
                {
                    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
                };

   using (var client = new HttpClient(handler))
   {
        client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
        client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate"));
        response = client.GetStringAsync("http://scooterlabs.com/echo").Result;
   }

AutomaticDecompression must be set in addition to the headers, otherwise you will get the compressed payload instead.