Python CLoader and Unicode

Question

When using CLoader as the Loader for yaml.load, it will not process unicode in my json. However, when I use the default loader the json is loaded fine.

I can work around this by catching the exception and loading it without CLoader.
Has anyone any experience with this that knows of a solution?

#!/usr/bin/python

import yaml
from yaml import CLoader as Loader1
from yaml import Loader
data_string = "\"nickname\": \"Stringhere \ud83d\udcaf\""
print "Loading with default loader"
json_data = yaml.load(data_string, Loader=Loader)
print "Loading with CLoader"
json_data = yaml.load(data_string, Loader=Loader1)



yaml.scanner.ScannerError: while parsing a quoted scalar
  in "<byte string>", line 1, column 13
found invalid Unicode character escape code
  in "<byte string>", line 1, column 27

Answer 1

See http://pyyaml.org/wiki/PyYAMLDocumentation

The PyYAML CLoader uses libyaml under the hood. And libyaml will reject such escapes. Only some \u escapes are supported, but not those which build a surrogate pair.

The PyYAML python parser (which is used by the loader imported with Loader), however, parses this, but still doesn't convert it to a unicode character.

The YAML 1.1/1.2 specifications don't explicitly mention surrogate pairs. However, since YAML 1.2 is a superset of JSON, and JSON supports this, I think libyaml should support this, too. (It targets YAML 1.1, but since it will currently simply return an error, it wouldn't hurt to add support for that.)

I created an issue for libyaml: https://github.com/yaml/libyaml/issues/110

https://en.wikipedia.org/wiki/UTF-16

The JSON RFC mentions surrogate pairs in Chapter 7: https://www.rfc-editor.org/rfc/rfc7159#section-7