htmlspecialchars outputting blank

Question

Using both htmlspecialchars and htmlentities is causing blank outputs from items such as a ™ symbol and even single ' quotes. Obviously, this is absolutely useless, however outputting the data without using html characters results in this symbol for both �. Any reason why this is occuring?

here is the code that is causing the problem:

<p>
<?php 
    echo nl2br(htmlspecialchars($aboutarray[0]['about_us'], ENT_COMPAT, "UTF-8")); 
?>
</p>

Answer 1

That string is not encoded in valid UTF-8 encoding. It could be in another encoding like UTF-16 or perhaps it just contains some binary junk that doesn't correspond to any format.

The bottom line is that, since you specified "UTF-8" as the encoding type parameter of htmlspecialchars(), it will return an empty string if the string does not comply with "UTF-8". It states this in the PHP manual.

A simple fix is to use the substitute or ignore flag. Change:

htmlspecialchars($aboutarray[0]['about_us'], ENT_COMPAT, "UTF-8")

To:

htmlspecialchars($aboutarray[0]['about_us'], ENT_COMPAT|ENT_SUBSTITUTE, "UTF-8")

Or:

htmlspecialchars($aboutarray[0]['about_us'], ENT_COMPAT|ENT_IGNORE, "UTF-8")

Note: ENT_IGNORE removes the non-compliant bytes. This could cause a security issue. It's better to truly understand the contents of your string and how it's being encoded. Correct the source of the problem rather than use the simple ENT_IGNORE fix.

You should ask yourself why your string is not encoded in UTF-8... it should be, but it's not.

I happen to have just encountered this problem as well; you can read details on why an empty string is being returned here.