HttpClient: Correct order to detect encoding

Question

I'm using HttpClient to fetch some files. I put the content into a byte array (bytes). Now I need to detect the encoding. The contenttype will be either html, css, JavaScript or XML contenttype.

Currently I check the charset from headers, then check for a BOM (byte order mark) before I finally check the first part of the file for a charset meta tag. Normally this works fine, because there are no conflicts.

But: Is that order correct (in case of conflict)?

The code I corrently use:

Encoding encoding;
try
{
    encoding = Encoding.GetEncoding(responseMessage.Content.Headers.ContentType.CharSet);
}
catch
{
    using (MemoryStream ms = new MemoryStream(bytes))
    {
        using (StreamReader sr = new StreamReader(ms, Encoding.Default, true))
        {
            char[] chars = new char[1024];
            sr.Read(chars, 0, 1024);
            string textDefault = new string(chars);
            if (sr.CurrentEncoding == Encoding.Default)
            {
                encoding = Global.EncodingFraContentType(textDefault);
            }
            else
            {
                encoding = sr.CurrentEncoding;
            }
        }
    }
}
responseInfo.Text = encoding.GetString(bytes);

Global.EncodingFraContentType is a regular expression that finds the charset defined either in XML declaration, or in a meta tag.

What order is the correct to detect charset/encoding?

Answer 1

The correct answer depends not on order, but on which actually gives the correct result, and there's no perfect answer here.

If there is a conflict, then the server has given you something incorrect. Since it's incorrect there can't be a "correct" order because there isn't a correct way of being incorrect. And, maybe the header and the embedded metadata are both wrong!

No even slightly common-used encoding can have something that looks like a BOM would look like in UTF-8 or UTF-16 at the beginning and still be a valid example of the content types you mention, so if there's a BOM then that wins.

(The one exception to that is if the document is so badly edited as to switch encoding part-way through, which is no unheard of, but then the buggy content is so very buggy as to have no real meaning).

If the content contains no octet that is greater than 0x7F then it doesn't matter and the header and metadata both claim it as different examples of US-ASCII, UTF-8, any of the ISO-8859 family of encodings, or any of the other encodings for which those octets all map to the same code point, then it doesn't really matter which you consider it to be, as the nett result is the same. Consider it to be whatever the metadata says, as then you don't need to rewrite it to match correctly.

If it's in UTF-16 without a BOM it is likely going to be clearly as such very soon as all of those formats have a lot of characters with special meaning in the range U+0000 to U+00FF (indeed, generally U+0020 to U+007F) and so you'll have lots of ranges with a zero byte every other character.

If it has octets above 0x7F and is valid UTF-8, then it's almost certainly UTF-8. (By the same token if it's not UTF-8 and has octets above 0x7F then it almost certainly can't be mistaken for UTF-8).

The trickiest reasonably common case is if you have conflicting claims about it being in two different encodings which are both single-octet-per-character encodings and an octet in the range 0x80-0xFF is present. This is the case that you can't be sure about. If one encoding is a subset of the other (especially when C1 controls are excluded) then you could go for the superset, but that requires storing knowledge about those encodings, and considerable amount of work. Most of the time I'd be inclined to just throw an exception, and when it's found in the logs see if I can get the source to fix their bug, or special-case that source, but that doesn't work if you are dealing with a very large number of disparate sources that you may not have a relationship with. Alas there is no perfect answer here.

Its worth noting also that sometimes both header and embedded metadata will agree with each other incorrectly. A common case is content in CP-1252 but claimed as being in ISO-8859-1.