preg_replace to strip out non-printing characters seems to remove all foreign characters as well

Question

I'm using the following regex to strip out non-printing control characters from user input before inserting the values into the database.

 preg_replace('/[\x00-\x1F\x80-\xFF]/', '', $value)

Is there a problem with using this on utf-8 strings? It seems to remove all non-ascii characters entirely.

Answer 1

Part of the problem is that you aren't treating the target as a UTF-8 string; you need the /u modifier for that. Also, in UTF-8 any non-ASCII character is represented by two or more bytes, all of them in the range \x80..\xFF. Try this:

preg_replace('/\p{Cc}+/u', '', $value)

\p{Cc} is the Unicode property for control characters, and the u causes both the regex and the target string to be treated as UTF-8.

Answer 2

You can use Unicode character properties

preg_replace('/[^\p{L}\s]/u','',$value);

(Do add the other classes you want to let through)

If you want to revert unicode to ascii, by no means fullproof but with some nice translations:

echo iconv('utf-8','ascii//translit','éñó'); //prints 'eno'