Why call mb_convert_encoding to sanitize text?

Question

This is in reference to this (excellent) answer. He states that the best solution for escaping input in PHP is to call mb_convert_encoding followed by html_entities.

But why exactly would you call mb_convert_encoding with the same to and from parameters (UTF8)?

Excerpt from the original answer:

Even if you use htmlspecialchars($string) outside of HTML tags, you are still vulnerable to multi-byte charset attack vectors.

The most effective you can be is to use the a combination of mb_convert_encoding and htmlentities as follows.

$str = mb_convert_encoding($str, 'UTF-8', 'UTF-8');
$str = htmlentities($str, ENT_QUOTES, 'UTF-8');

Does this have some sort of benefit I'm missing?

Answer 1

Not all binary data is valid UTF8. Invoking mb_convert_encoding with the same from/to encodings is a simple way to ensure that one is dealing with a correctly encoded string for the given encoding.

A way to exploit the omission of UTF8 validation is described in section 6 (security considerations) in rfc2279:

Another example might be a parser which prohibits the octet sequence 2F 2E 2E 2F ("/../"), yet permits the illegal octet sequence 2F C0 AE 2E 2F.

This may be more easily understood by examining the binary representation:

110xxxxx 10xxxxxx # header bits used by the encoding
11000000 10101110 # C0 AE
         00101110 #    2E the '.' character

In other words: (C0 AE - header-bits) == '.'

As the quoted text points out, C0 AE is not a valid UTF8 octet sequence, so mb_convert_encoding would have removed it from the string (or translated it to '.', or something else :-).