compression (header) bug in .NET HttpWebRequest?

Question

I just noticed that using the .NET HttpWebRequest, I wasn't getting gzip'd contents back from the server (running nginx 0.8.52). Testing with curl, I verified that gzip was set up properly server-side, then dug into the issue with the VS debugger.

The culprit turns out to be how the Accept-Encoding header field is generated: if I set

 request.AutomaticDecompression = DecompressionMethods.GZip | 
                                   DecompressionMethods.Deflate`

I get the following header:

`Accept-Encoding: gzip, deflate,gzip, deflate`. 

If I set

request.AutomaticDecompression = DecompressionMethods.GZip; 

I get

Accept-Encoding: gzip,gzip

I haven't checked what the HTTP specification says, but nginx ought to handle the situation, but instead it returns Vary: Accept-Encoding back. On the other hand, the Accept-Encoding header generated by HttpWebRequest definitely doesn't look right, and seems like a bug to me.

If I don't specify AutomaticDecompression and manually set the Accept-Encoding header, I get gzip'd content back, but HttpWebRequest seems to be pretty dumb, and doesn't decode the compressed stream (I guess it relies on an internal IsCompressed flag rather than parsing the response headers).

Any suggestions?

EDIT:

It should be noted that there's authentication involved; I've just found out that HttpWebRequest initially does a request without including the supplied credentials. For this request, the Accept-Encoding header is specified correctly. The duplication occurs after getting the 401 server response and re-doing the request with credentials.

EDIT 2:

I posted a Microsoft Connect bug report.

Answer 1

Seems to be a bug alright. Possible workarounds:

1. Don't use AutomaticDecompression, set "Accept-Encoding" header field manually, and manually handle decompression of the response stream if "Content-Encoding" property is set.
2. Don't use Credentials property, instead manually construct HTTP Authorization header. This gets rid of unnecessary server roundtrip and fixes the header field duplication bug.