How to replace/escape U+2028 or U+2029 characters in PHP to stop my JSONP API breaking

Question

Ok I am running a public JSONP API which the data is served from my PHP server. I just read this article:

• JSON: The JavaScript subset that isn't (by Magnus Holm; May 2011)
  (please read for clarification)

Basically if my JSON strings contains a U+2028 character (Unicode line separator) or U+2029 character (Unicode paragraph separator) then this is perfectly valid JSON. However when using JSONP the JSON gets executed as JavaScript and no string in JavaScript can contain a literal U+2028 or a U+2029 as it will break the JavaScript. Apparently this is usually not a problem as long as you use a proper JSON parser, but in the case of JSONP the browser is the JSON parser.

Essentially if these characters were inside strings in my JSONP data being sent to the client this would throw a line or paragraph break into the string which would break the JavaScript and stop it executing. This is a possibility as the API is sending back some client entered data. Someone could potentially enter a U+2028 or a U+2029 into the database, so when I send that back as JSONP it will break any implementation using my API.

So my question is, in PHP how can I sanitise/output escape the JSON data to remove or escape the U+2028 and U+2029 characters before sending it to the client?

Currently my process is doing a json_encode on an array of data and sending that data down to the client. Should I escape the data by looping through the array and filtering it, or escape all the JSON encoded string all at once?

The other thing is I'm not sure how to escape the U+2028 and U+2029 characters in PHP anyway. Can I just do a str_replace? I'm not sure if str_replace is multibyte safe and there's no mb_str_replace function unless I use some custom made one. So how do you remove/escape those unicode characters?

Thanks very much.

Answer 1

You can replace U+2028, U+2029 with "\u2028", "\u2029" either on the PHP side or the JavaScript side, or both, it doesn't matter as long as it happens at least once (it's idempotent).

You can just use ordinary string replacement functions. They don't need to be "multibyte safe", and you can do it just as easily in any Unicode encoding (UTF-8, UTF-16, UTF-32 are all equally fine). PHP didn't have Unicode escape sequences last time I checked which is just one more reason why PHP is a joke but you can use the \x escape with UTF-8...

(In short, the reason there's no multibyte string replace function is because it would be redundant -- it would be exactly the same as a non-multibyte string replace function.)

// Javascript
data = data.replace("\u2028", "\\u2028").replace("\u2029", "\\u2029");

// PHP
$data = str_replace("\xe2\x80\xa8", '\\u2028', $data);
$data = str_replace("\xe2\x80\xa9", '\\u2029', $data);

Or you could just do nothing at all, since PHP escapes non-Unicode characters by default in json_encode():

// Safe
echo json_encode("\xe2\x80\xa9");
--> "\u2029"

// Correct JSON, but invalid Javascript...
// (Well, technically, JSON root must be array or object)
echo json_encode("\xe2\x80\xa9", JSON_UNESCAPED_UNICODE);
--> "
"

Answer 2

It’s worth pointing out that this is no longer necessary.

By default, json_encode() encodes all non-ASCII characters (including U+2028 & U+2029), and also escapes the forward slash, even though that does not need to be escaped by the JSON spec. It does no harm to escape it, and it can be safer in certain contexts. So, by default, these characters are escaped anyway.

The JSON_UNESCAPED_UNICODE constant outputs unescaped Unicode, which can save bytes. However, just as the slash character is escaped because it can be dangerous in some contexts, so too U+2028 & U+2029 are also escaped, because they too are dangerous in some contexts. This was not the case at the time you asked your question: this feature has been added to PHP more recently.

(These extra escapes can be turned off with JSON_UNESCAPED_SLASHES and JSON_UNESCAPED_LINE_TERMINATORS, respectively.)