What is the difference between u' ' prefix and unicode() in python?

Question

What is the difference between u'' prefix and unicode()?

# -*- coding: utf-8 -*-
print u'上午'  # this works
print unicode('上午', errors='ignore') # this works but print out nothing
print unicode('上午') # error

For the third print, the error shows: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 0

If I have a text file containing non-ascii characters, such as "上午", how to read it and print it out correctly?

Answer 1

• u'..' is a string literal, and decodes the characters according to the source encoding declaration.

• unicode() is a function that converts another type to a unicode object, you've given it a byte string literal. It'll decode a byte string according to the default ASCII codec.

So you created a byte string object using a different type of literal notation, then tried to convert it to a unicode() object, which fails because the default codec for str -> unicode conversions is ASCII.

The two are quite different beasts. If you want to use the latter, you need to give it an explicit codec:

print unicode('上午', 'utf8')

The two are related in the same way that using 0xFF and int('0xFF', 0) are related; the former defines an integer of value 255 using hex notation, the latter uses the int() function to extract an integer from a string.

An alternative method would be to use the str.decode() method:

print '上午'.decode('utf8')

Don't be tempted to use an error handler (such as ignore' or 'replace') unless you know what you are doing. 'ignore' especially can mask underlying issues with having picked the wrong codec, for example.

You may want to read up on Python and Unicode:

• Pragmatic Unicode by Ned Batchelder

• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky

• The Python Unicode HOWTO

Answer 2

When a str is not prefixed by u'' in Python 2.7.x, what the interpreter sees is a byte string, without an explicit encoding.

If you do not tell the interpreter what to do with those bytes when executing unicode(), it will (as you saw) default to trying to decode the bytes it sees via the ascii encoding scheme.

It does so as a preliminary step in trying to turn the plain bytes of the str into a unicode object.

Using ascii to decode means: try to interpret each byte of the str using a hard-coded mapping, a number between 0 and 127.

The error you encountered was like a dict KeyError: the interpreter encountered a byte for which the ascii encoding scheme does not have a specified mapping.

Since the interpreter doesn't know what to do with the byte, it throws an error.

You can change that preliminary step by telling the interpreter to decode the bytes using another set of encoding/decoding mappings instead, one that goes beyond ascii, such as UTF-8, as elaborated in other answers.

If the interpreter finds a mapping in the chosen scheme for each byte (or bytes) in the str, it will decode successfully, and the interpreter will use the resulting mappings to produce a unicode object.

A Python unicode object is a series of Unicode code points. There are 1,112,064 valid code points in the Unicode code space.

And if the scheme you choose for decoding is the one with which your text (or code points) were encoded, then the output when printing should be identical to the original text.

Can also consider trying Python 3. The relevant difference is explained in the first comment below.