Writing pandas DataFrame to JSON in unicode

Question

I'm trying to write a pandas DataFrame containing unicode to json, but the built in .to_json function escapes the characters. How do I fix this?

Example:

import pandas as pd df = pd.DataFrame([['τ', 'a', 1], ['π', 'b', 2]]) df.to_json('df.json') 

This gives:

{"0":{"0":"\u03c4","1":"\u03c0"},"1":{"0":"a","1":"b"},"2":{"0":1,"1":2}} 

Which differs from the desired result:

{"0":{"0":"τ","1":"π"},"1":{"0":"a","1":"b"},"2":{"0":1,"1":2}} 

---

I have tried adding the force_ascii=False argument:

import pandas as pd df = pd.DataFrame([['τ', 'a', 1], ['π', 'b', 2]]) df.to_json('df.json', force_ascii=False) 

But this gives the following error:

UnicodeEncodeError: 'charmap' codec can't encode character '\u03c4' in position 11: character maps to <undefined> 

---

I'm using WinPython 3.4.4.2 64bit with pandas 0.18.0

Answer 1

Opening a file with the encoding set to utf-8, and then passing that file to the .to_json function fixes the problem:

with open('df.json', 'w', encoding='utf-8') as file:     df.to_json(file, force_ascii=False) 

gives the correct:

{"0":{"0":"τ","1":"π"},"1":{"0":"a","1":"b"},"2":{"0":1,"1":2}} 

Note: it does still require the force_ascii=False argument.