Using UTF-8 encoded JSON fixture file in Django

Question

I'm trying to write a JSON initial data fixture that will be loaded after every call to syncdb.

I placed an initial_data.json file in my mysite/myapp/fixtures directory:

[
  {
    "model": "myapp.Person",
    "pk": 1,
    "fields": {
      "first_name": "Tom",
      "last_name": "Yam"
    }
  }
]

Everything is working when the file is encoded in ASCII, but when I save it in UTF-8 encoding (I need to use non-ASCII characters) I get to following error:

Problem installing fixture 'initial_data.json': Traceback (most recent call last):
File "D:\Tom\DjangoEnv\Lib\site-packages\django\core\management\commands\loaddata.py", line 190, in handle
for obj in objects:
File "D:\Tom\DjangoEnv\Lib\site-packages\django\core\serializers\json.py", line 47, in Deserializer
raise DeserializationError(e)
DeserializationError: No JSON object could be decoded

According to the Django documentation, I need to set ensure_ascii=False when working with non-ASCII data and JSON serializers, but I can't figure how to do it (since its being called from the syncdb function.

Any ideas how to use a UTF-8 encoded JASON file as a fixture?

Answer 1

load_data would not pass ensure_ascii option to serializer so you have two options:

1. convert data to ascii unicode escaped before loading it, ie:

   import codecs
   encoded = codecs.open('/tmp/tst.txt', 'r', 'utf-8').read().encode(
               'ascii', 'backslashreplace')
   open('/tmp/tst-encoded.txt', 'w').write(encoded)
   

2. write your own management command that would pass ensure_ascii

hope this helps.