Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How should escaped unicode be handled by json parsers and encoders?

Tags:

json

unicode

The json spec allows for escaped unicode in json strings (of the form \uXXXX). It specifically mentions a restricted codepoint (a noncharacter) as a valid escaped codepoint. Doesn't this imply parsers should generate illegal unicode from strings containing noncharacters and restricted codepoints?

An example:

{ "key": "\uFDD0" }

decoding this either requires your parser makes no attempt to interpret the escaped codepoint or that it generates an invalid unicode string. does it not?

like image 349
ArgumentError Avatar asked Oct 04 '09 04:10

ArgumentError


People also ask

Does JSON allow Unicode?

The JSON specification states that JSON strings can contain unicode characters in the form of: "here comes a unicode character: \u05d9 !"

What encoding is used for JSON?

The default encoding is UTF-8. (in §6) JSON may be represented using UTF-8, UTF-16, or UTF-32. When JSON is written in UTF-8, JSON is 8bit compatible. When JSON is written in UTF-16 or UTF-32, the binary content-transfer-encoding must be used.

Is JSON UTF-8 or UTF 16?

JSON text SHALL be encoded in Unicode. The default encoding is UTF-8.


2 Answers

When you decode, it seems that this would be an appropriate use for the unicode replacement character, U+FFFD.

From the Unicode Character Database:

  • used to replace an incoming character whose value is unknown or unrepresentable in Unicode
  • compare the use of U+001A as a control character to indicate the substitute function
like image 153
Adam Goode Avatar answered Nov 15 '22 23:11

Adam Goode


What do you mean by “restricted codepoint”? What spec are you looking at that uses that language? (I can't find any such.)

If you are talking about the surrogates then yes: JavaScript knows almost nothing(*) about surrogates and treats all UTF-16 codepoints in any sequence as valid. JSON, being limited to what JavaScript supports, does the same.

*: the only part of JS I can think of that does anything special with surrogates is the encodeURIComponent function, as it uses UTF-8 encoding, in which an attempt to encode an invalid surrogate sequence cannot work. If you try to:

encodeURIComponent('\ud834\udd1e'.substring(0, 1))

you will get an exception.

(Gah! SO seems not to allow characters from outside the Basic Multilingual Plane to be posted directly. Tsk.)

like image 33
bobince Avatar answered Nov 15 '22 21:11

bobince