Using json loads produces 'ascii codec can't encode characters' error in Python 3.6 [closed]

Question

It feels like a a very basic question and I've already read through the docs and answers, that suggest my code should be working. This may really be a duplicate and I missed something, please, drop a link as I've already spent a couple of hours on this and it feels silly. Thank you in advance.

Python 3.6 code:

import json
print( json.loads( '{"text": \"\\u0444\\u044b\\u0432\\u0430\"}' ) )

Produces the following error:

Traceback (most recent call last):
  File "test2.py", line 28, in <module>
    print( json.loads( '{"txt": \"\\u0444\\u044b\\u0432\\u0430\"}' ) )
UnicodeEncodeError: 'ascii' codec can't encode characters in position 9-12: ordinal not in range(128)

I had an impression that I can do

jsn = json.dumps( my_dict )
# and later call
json.loads( jsn )

And I will get back my original dict, but this doesn't work for unicode characters.

Can I convert a string containing unicode characters back to python dict with json.loads ? Please, provide a minimum working code for my example.

Answer 1

The problem is not your code but your encoding settings of stdout.

The reason is the same as the following code:

import json
print(str(json.loads( '{"text": \"\\u0444\\u044b\\u0432\\u0430\"}' )).encode('ascii'))

print use stdout for default output, so your encoding settings of stdout is not "utf-8". To check the encoding settings, type following code

import sys; print(sys.stdout)

you will get something like this:

<_io.TextIOWrapper name='' mode='w' encoding='ANSI_X3.4-1968'>

the content of encoding is not utf-8; to change that, reopen stdout with utf-8 encoding.

import sys
import io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="UTF-8")

now the print should work.

but the deeper solution for this problem is to correct the environment settings of your system.

I have a solution for linux. type locale -a in your terminal, the language settings of your current system will be listed, let we choose a "utf-8" encoded language.

export LANG=en_US.UTF-8 # for english user

or

export LANG=zh_CN.UTF-8 # 给中文用户

you may need to add it into .bashrc to make sure it always effective.