Python 2.7 JSON dump UnicodeEncodeError

Question

I have a file where each line is a json object like so:

{"name": "John", ...}

{...}

I am trying to create a new file with the same objects, but with certain properties removed from all of them.

When I do this, I get a UnicodeEncodeError. Strangely, If I instead loop over range(n) (for some number n) and use infile.next(), it works just as I want it to.

Why so? How do I get this to work by iterating over infile? I tried using dumps() instead of dump(), but that just makes a bunch of empty lines in the outfile.

with open(filename, 'r') as infile:
    with open('_{}'.format(filename), 'w') as outfile:
        for comment in infile:
            decodedComment = json.loads(comment)
            for prop in propsToRemove:
                # use pop to avoid exception handling
                decodedComment.pop(prop, None)
            json.dump(decodedComment, outfile, ensure_ascii = False)
            outfile.write('\n')

Here is the error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\U0001f47d' in position 1: ordinal not in range(128)

Thanks for the help!

Answer 1

The problem you are facing is that the standard file.write() function (called by the json.dump() function) does not support unicode strings. From the error message, it turns out that your string contains the UTF character \U0001f47d (which turns out to code for the character EXTRATERRESTRIAL ALIEN, who knew?), and possibly other UTF characters. To handle these characters, either you can encode them into an ASCII encoding (they'll show up in your output file as \XXXXXX), or you need to use a file writer that can handle unicode.

To do the first option, replace your writing line with this line:

json.dump(unicode(decodedComment), outfile, ensure_ascii = False)

The second option is likely more what you want, and an easy option is to use the codecs module. Import it, and change your second line to:

with codecs.open('_{}'.format(filename), 'w', encoding="utf-8") as outfile:

Then, you'll be able to save the special characters in their original form.