HttpRequestHeader Content encoding issue

Question

I am using below code snippet to download HTTP response to local file. Sometimes my content which is in url is multi-lingual (chinese, japanese, thai data etc.). I am using ContentEncoding header to specify my content is in UTF-8 encoding, but this has no effect in my local output file which is generating in ASCII. Due to this, multi-lingual data is corrupted. Any help?

using (var webClient = new WebClient())
        {
            webClient.Credentials = CredentialCache.DefaultCredentials;
            webClient.Headers.Add(HttpRequestHeader.UserAgent, "Mozilla/4.0");
            webClient.Headers.Add(HttpRequestHeader.ContentEncoding, "utf-8");

            webClient.DownloadFile(url, @"c:\temp\tempfile.htm");
        }

Answer 1

The ContentEncoding header is not used to specify the character set. It's used by the client to say what kind of encoding (compression) it supports.

The client can't tell the server what character set to send. The server sends its data and some header fields that say what character set is being used. Typically it's in the ContentTypeheader and looks like: text/html; charset=UTF-8.

When you're using WebClient, you want to set the Encoding property as a fallback so that if the server doesn't identify the character set, your default will be used. For example:

WebClient client = new WebClient();
client.Encoding = Encoding.UTF8;
string s = client.DownloadString(DownloadUrl);

See http://www.informit.com/guides/content.aspx?g=dotnet&seqNum=800 for a bit more information.