Bug in PHP Preg engine: look-around Unicode issue

Question

Why following js code words:

"آرد@".replace(/(?=.)/g,'!'); // returns: ""!آ!ر!د""

But its php equivalent returns '!�!�!�!�!�!�'?

preg_replace('/(?=.)/u', '!', 'آرد'); //returns '!�!�!�!�!�!�'

This works only in 4.3.5 - 5.0.5, 5.1.1 - 5.1.6 versions.

See: http://3v4l.org/jrV0W

Answer 1

If you simply add the /u modifier, the pattern is supposed to be treated as utf-8. The second example works because:

1. Since PHP 5.1, you can use \p{L} that can be translated as: "is any kind of letter from any language."
2. In addition to the standard notation, \p{L}, Java, Perl, PCRE and now PHP allow you to use the shorthand \pL. The shorthand only works with single-letter Unicode properties.

---

UPDATE: Why preg_replace('/(?=.)/u', '!', 'آرد'); //returns '!�!�!�!�!�!�'??

As @MarkFox says, the reason is because in the context of preg_replace() it assumes one byte per character and the characters you're "RegExing" are multibyte. That's why your replace output has double the matches you'd expect, it's matching each byte of each character (which I infer to be two bytes) -

No matter what you do with your document encoding, you will need to use Unicode character properties to get this working.

What about that weird symbol?

When you see that "weird square symbol with a question mark inside" otherwise known as the REPLACEMENT CHARACTER, that is usually an indicator that you have a byte in the range of 80-FF (128-255) and the system is trying to render it in UTF-8.

That entire byte-range is invalid for single-byte characters in UTF-8, but are all very common in western encodings such as ISO-8859-1.