Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What's the deal with Python 3.4, Unicode, different languages and Windows?

Tags:

python

unicode

Happy examples:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

czech = u'Leoš Janáček'.encode("utf-8")
print(czech)

pl = u'Zdzisław Beksiński'.encode("utf-8")
print(pl)

jp = u'リング 山村 貞子'.encode("utf-8")
print(jp)

chinese = u'五行'.encode("utf-8")
print(chinese)

MIR = u'Машина для Инженерных Расчётов'.encode("utf-8")
print(MIR)

pt = u'Minha Língua Portuguesa: çáà'.encode("utf-8")
print(pt)

Unhappy output:

b'Leo\xc5\xa1 Jan\xc3\xa1\xc4\x8dek'
b'Zdzis\xc5\x82aw Beksi\xc5\x84ski'
b'\xe3\x83\xaa\xe3\x83\xb3\xe3\x82\xb0 \xe5\xb1\xb1\xe6\x9d\x91 \xe8\xb2\x9e\xe5\xad\x90'
b'\xe4\xba\x94\xe8\xa1\x8c'
b'\xd0\x9c\xd0\xb0\xd1\x88\xd0\xb8\xd0\xbd\xd0\xb0 \xd0\xb4\xd0\xbb\xd1\x8f \xd0\x98\xd0\xbd\xd0\xb6\xd0\xb5\xd0\xbd\xd0\xb5\xd1\x80\xd0\xbd\xd1\x8b\xd1\x85 \xd0\xa0\xd0\xb0\xd1\x81\xd1\x87\xd1\x91\xd1\x82\xd0\xbe\xd0\xb2'
b'Minha L\xc3\xadngua Portuguesa: \xc3\xa7\xc3\xa1\xc3\xa0'

And if I print them like this:

jp = u'リング 山村 貞子'
print(jp)

I get:

Traceback (most recent call last):
  File "x.py", line 5, in <module>
    print(jp)
  File "C:\Python34\lib\encodings\cp850.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position
0-2: character maps to <undefined>

I've also tried the following from this question (And other alternatives that involve sys.stdout.encoding):

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from __future__ import print_function
import sys

def safeprint(s):
    try:
        print(s)
    except UnicodeEncodeError:
        if sys.version_info >= (3,):
            print(s.encode('utf8').decode(sys.stdout.encoding))
        else:
            print(s.encode('utf8'))

jp = u'リング 山村 貞子'
safeprint(jp)

And things get even more cryptic:

リング 山村 貞子

And the docs were not very helpful.

So, what's the deal with Python 3.4, Unicode, different languages and Windows? Almost all possible examples I could find, deal with Python 2.x.

Is there a general and cross-platform way of printing ANY Unicode character from any language in a decent and non-nasty way in Python 3.4?

EDIT:

I've tried typing at the terminal:

chcp 65001

To change the code page, as proposed here and in the comments, and it did not work (Including the attempt with sys.stdout.encoding)

like image 565
Ericson Willians Avatar asked May 29 '15 22:05

Ericson Willians


People also ask

Why does Python use unicode?

Python's string type uses the Unicode Standard for representing characters, which lets Python programs work with all these different possible characters. Unicode (https://www.unicode.org/) is a specification that aims to list every character used by human languages and give each character its own unique code.

Does Python 3 use Ascii or unicode?

Python 3 came and fixed this. Strings are still str type by default but they now mean unicode code points instead — we carry what we see. If we want to store these str type strings in files we use bytes type instead. Default encoding is UTF-8 instead of ASCII.

What is unicode in Python 3?

The Unicode standard describes how characters are represented by code points. For example the characters above are represented with the code points U+0041, U+0042, U+0043, and U+00C9, respectively. Basically, code points are numbers in the range from 0 to 0x10FFFF.

What does unicode mean in Python?

Unicode is also called Universal Character set. ASCII uses 8 bits(1 byte) to represents a character and can have a maximum of 256 (2^8) distinct combinations.


2 Answers

Update: Since Python 3.6, the code example that prints Unicode strings directly should just work now (even without py -mrun).


Python can print text in multiple languages in Windows console whatever chcp says:

T:\> py -mpip install win-unicode-console
T:\> py -mrun your_script.py

where your_script.py prints Unicode directly e.g.:

#!/usr/bin/env python3
print('š áč')      # cz
print('ł ń')       # pl
print('リング')     # jp
print('五行')      # cn
print('ш я жх ё') # ru
print('í çáà')    # pt

All you need is to configure the font in your Windows console that can display the desired characters.

You could also run your Python script via IDLE without installing non-stdlib modules:

T:\> py -midlelib -r your_script.py

To write to a file/pipe, use PYTHONIOENCODING=utf-8 as @Mark Tolonen suggested:

T:\> set PYTHONIOENCODING=utf-8
T:\> py your_script.py >output-utf8.txt 

Only the last solution supports non-BMP characters such as 😒 (U+1F612 UNAMUSED FACE) -- py -mrun can write them but Windows console displays them as boxes even if the font supports corresponding Unicode characters (though you can copy-paste the boxes into another program, to get the characters).

like image 161
jfs Avatar answered Oct 05 '22 23:10

jfs


The problem iswas (see Python 3.6 update below) with the Windows console, which supports an ANSI character set appropriate for the region targeted by your version of Windows. Python throws an exception by default when unsupported characters are output.

Python can read an environment variable to output in other encodings, or to change the error handling default. Below, I've read the console default and change the default error handling to print a ? instead of throwing an error for characters that are unsupported in the console's current code page.

C:\>chcp
Active code page: 437   # Note, US Windows OEM code page.

C:\>set PYTHONIOENCODING=437:replace

C:\>example.py
Leo? Janá?ek
Zdzis?aw Beksi?ski
??? ?? ??
??
?????? ??? ?????????? ????????
Minha Língua Portuguesa: çáà

Note the US OEM code page is limited to ASCII and some Western European characters.

Below I've instructed Python to use UTF8, but since the Windows console doesn't support it, I redirect the output to a file and display it in Notepad:

C:\>set PYTHONIOENCODING=utf8
C:\>example >out.txt
C:\>notepad out.txt

enter image description here

On Windows, its best to use a Python IDE that supports UTF-8 instead of the console when working with multiple languages. If only using one language, select it as the system locale in the Region and Language control panel and the console will support the characters of that language.

Update for Python 3.6

Python 3.6 now uses Windows Unicode APIs to write directly to the console, so the only limit is the console font's support of the characters. The following code works in a US Windows console. I have a Chinese language pack installed, it even displays the Chinese and Japanese if the console font is changed. Even without the correct font, replacement characters are shown in the console. Cut-n-paste to an environment such as this web page will display the characters correctly.

#!python3.6
#coding: utf8
czech = 'Leoš Janáček'
print(czech)

pl = 'Zdzisław Beksiński'
print(pl)

jp = 'リング 山村 貞子'
print(jp)

chinese = '五行'
print(chinese)

MIR = 'Машина для Инженерных Расчётов'
print(MIR)

pt = 'Minha Língua Portuguesa: çáà'
print(pt)

Output:

Leoš Janáček
Zdzisław Beksiński
リング 山村 貞子
五行
Машина для Инженерных Расчётов
Minha Língua Portuguesa: çáà
like image 20
Mark Tolonen Avatar answered Oct 06 '22 00:10

Mark Tolonen