UnicodeEncodeError: 'ascii' codec can't encode character in position 0: ordinal not in range(128)

Question

I'm working on a Python script that uses the scissor character (9986 - ✂) and I'm trying to port my code to Mac, but I'm running into this error.

The scissor character shows up fine when run from IDLE (Python 3.2.5 - OS X 10.4.11 iBook G4 PPC) and the code works entirely fine on Ubuntu 13.10, but when I attempt to run this in the terminal I get this error/traceback:

Traceback (most recent call last):   File "snippets-convert.py", line 352, in <module>     main()   File "snippets-convert.py", line 41, in main     menu()   File "snippets-convert.py", line 47, in menu     print ("|\t ",snipper.decode(),"PySnipt'd",snipper.decode(),"\t|") UnicodeEncodeError: 'ascii' codec can't encode character '\u2702' in position 0: ordinal not in range(128) 

and the code that is giving me the problem:

print ("|\t ",chr(9986),"PySnipt'd",chr(9986),"\t|")

Doesn't this signal that the terminal doesn't have the capability to display that character? I know this is an old system, but it is currently the only system I have to use. Could the age of the OS is interfering with the program?

I've read over these questions:

• UnicodeEncodeError: 'ascii' codec can't encode character u'\xef' in position 0: ordinal not in range(128) - Different character

• "UnicodeEncodeError: 'ascii' codec can't encode character" - Using 2.6, so don't know if it applies

• UnicodeEncodeError: 'ascii' codec can't encode character? - Seems to be a plausible solution to my problem, .encode('UTF-8'), I don't get the error. However, it displays a character code, not the character I want, and .decode() just gives me the same error. Not sure if I'm doing this right.

• UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-6: ordinal not in range(128) - Not sure if this applies, he's using a GUI, getting input, and all in Greek.

What's causing this error? Is it the age of the system/OS, the version of Python, or some programming error?

EDIT: This error crops up later with this duplicate issue (just thought I'd add it as it is within the same program and is the same error):

Traceback (most recent call last):   File "snippets-convert.py", line 353, in <module>     main()   File "snippets-convert.py", line 41, in main     menu()   File "snippets-convert.py", line 75, in menu     main()   File "snippets-convert.py", line 41, in main     menu()   File "snippets-convert.py", line 62, in menu     search()   File "snippets-convert.py", line 229, in search     print_results(search_returned)      # Print the results for the user   File "snippets-convert.py", line 287, in print_results     getPath(toRead)                                             # Get the path for the snippet   File "snippets-convert.py", line 324, in getPath     snipXMLParse(path)   File "snippets-convert.py", line 344, in snipXMLParse     print (chr(164),child.text) UnicodeEncodeError: 'ascii' codec can't encode character '\xa4' in position 0: ordinal not in range(128) 

EDIT:

I went into the terminal character settings and it does in fact support that character (as you can see in this screenshot:

when I insert it into terminal it prints out this: \342\234\202 and when I press Enter I get this: -bash: ✂: command not found

EDIT Ran commands as @J.F. Sebastian asked:

python3 test-io-encoding.py:

PYTHONIOENCODING:       None locale(False):  US-ASCII device(stdout): US-ASCII stdout.encoding:        US-ASCII device(stderr): US-ASCII stderr.encoding:        US-ASCII device(stdin):  US-ASCII stdin.encoding: US-ASCII locale(False):  US-ASCII locale(True):   US-ASCII 

python3 -S test-io-encoding.py:

PYTHONIOENCODING:       None locale(False):  US-ASCII device(stdout): US-ASCII stdout.encoding:        US-ASCII device(stderr): US-ASCII stderr.encoding:        US-ASCII device(stdin):  US-ASCII stdin.encoding: US-ASCII locale(False):  US-ASCII locale(True):   US-ASCII 

EDIT Tried the "hackerish" solution provided by @PauloBu:

As you can see, this caused one (Yay!) scissor, but I am now getting a new error. Traceback/error:

+-=============================-+ ✂Traceback (most recent call last):   File "snippets-convert.py", line 357, in <module>     main()   File "snippets-convert.py", line 44, in main     menu()   File "snippets-convert.py", line 52, in menu     print("|\t "+sys.stdout.buffer.write(chr(9986).encode('UTF-8'))+" PySnipt'd "+ sys.stdout.buffer.write(chr(9986).encode('UTF-8'))+" \t|") TypeError: Can't convert 'int' object to str implicitly 

EDIT Added results of @PauloBu's fix:

+-=============================-+ | ✂ PySnipt'd  ✂       | +-=============================-+ 

EDIT:

And his fix for his fix:

+-=============================-+ ✂✂|       PySnipt'd     | +-=============================-+ 

Answer 1

When Python prints and output, it automatically encodes it to the target medium. If it is a file, UTF-8 will be used as default and everyone will be happy, but if it is a terminal, Python will figure out the encoding the terminal is using and will try to encode the output using that one.

This means that if your terminal is using ascii as encoding, Python is trying to encode scissor char to ascii. Of course, ascii doesn't support it so you get Unicode decode error.

This is why you always have to explicitly encode your output. Explicit is better than implicit remember? To fix your code you may do:

import sys sys.stdout.buffer.write(chr(9986).encode('utf8')) 

This seems a bit hackerish. You can also set PYTHONIOENCODING=utf-8 before executing the script. I'am uncomfortable with both solutions. Probably your console doesn't support utf-8 and you see gibberish. But your program will be behaving correctly.

What I strongly recommend if you definitely need to show correct output on your console is to set your console to use another encoding, one that support scissor character. (utf-8 perhaps). On Linux, that can be achieve by doing: export lang=UTF_8. On Windows you change the console's code page with chcp. Just figure out how to set utf8 in yours and IMHO that'll be the best solution.

---

You can't mix print and sys.stdout.write because they're basically the same. Regarding to your code, the hackerish way would be like this:

sys.stdout.buffer.write(("|\t "+ chr(9986) +" PySnipt'd " + chr(9986)+" \t|").encode('utf8')) 

I suggest you to take a read at the docs to see what's going on under the hood with print function and with sys.stdout: http://docs.python.org/3/library/sys.html#sys.stdin

Hope this helps!

Answer 2

test_io_encoding.py output suggests that you should change your locale settings e.g., set LANG=en_US.UTF-8.

---

The first error might be due to you are trying to decode a string that is already Unicode. Python 2 tries to encode it using a default character encoding ('ascii') before decoding it using (possibly) different character encoding. The error happens on the encode step:

>>> u"\u2702".decode() # Python 2 Traceback (most recent call last):   File "<stdin>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode character u'\u2702' in position 0: ordinal not in range(128) 

It looks like you are running your script using Python 2 instead of Python 3. You would get:

>>> "\u2702".decode() # Python 3 Traceback (most recent call last):   File "<stdin>", line 1, in <module> AttributeError: 'str' object has no attribute 'decode' 

different error otherwise.

Just drop the .decode() call:

print("|\t {0} PySnipt'd {0} \t|".format(snipper)) 

The second issue is due to printing a Unicode string into a pipe:

$ python3 -c'print("\u2702")' ✂ $ python3 -c'print("\u2702")' | cat Traceback (most recent call last):   File "<string>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode character '\u2702' in position 0: ordinal not in range(128) 

Set appropriate for your purpose PYTHONIOENCODING environment variable:

$ PYTHONIOENCODING=utf-8 python3 -c'print("\u2702")' | cat ✂ 

---

the terminal is just displaying this: | b'\xe2\x9c\x82' PySnipt'd b'\xe2\x9c\x82' |

If snipper is a bytes object then leave the snipper.decode() calls.

$ python3 -c"print(b'\xe2\x9c\x82'.decode())" ✂ $ python3 -c"print(b'\xe2\x9c\x82'.decode())" | cat Traceback (most recent call last):   File "<string>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode character '\u2702' in position 0: ordinal not in range(128) 

The fix is the same:

$ PYTHONIOENCODING=utf-8 python3 -c"print(b'\xe2\x9c\x82'.decode())" | cat ✂