Fix Python Unicode Error caused by another language

Question

I'm getting this error:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 52-57: ordinal not in range(128)

The code causing the error:

f.write(textwrap.dedent(unicode(the_string))

The string I'm trying to write is an excerpt from a website which contains English and Japanese (for testing) Transliteration of foreign words and names, such as コンピュータ (konpyūta, "computer") and ロンドン (Rondon, "London"). (Some foreign borrowings that have become naturalized may not be rendered in katakana.)

When handling a string in Python that is In another language (Japanese). How can I parse this to prevent the error and still retain the string?

Answer 1

The problem here is that the .write method of the file object is naively trying to convert the unicode string (if you are in Python 2.x, this will be of the unicode type) that you pass it to a byte string (if you are in Python 2.x, this will be of the str type) using the ASCII codec, but the unicode string you are passing cannot be represented using ASCII because it has (Japanese) characters that are not part of the ASCII character set.

You need to use the .encode method of the unicode string to convert it to a series of bytes that represent that string before you can save it. That's basically what the str type represents in Python 2.x - just a series of bytes, not a series of characters like you might expect. Python can easily make you think otherwise, though, because when you print a variable of type str, Python displays it - as a series of characters in your terminal - using the system's default unicode encoding.

What encoding you should use here to encode your string depends upon your use case. UTF-8 is the most common and you may simply want to use that, but if you want to make sure that the file you are writing to will display properly in a text editor on the same system even if you run it on a device with a less usual system encoding like UTF-16, you may instead want to use the default encoding of the system (which will of course fail if the system has a default encoding that cannot encode your string).

In other words, you almost certainly want to do one of the following things:

a)

f.write(textwrap.dedent(the_string).encode('utf-8'))

b)

import sys
f.write(textwrap.dedent(the_string).encode(sys.getdefaultencoding()))

If you think that this is a pretty irritating and complicated load of stuff to get your head around to perform the fairly basic task of writing some non-ASCII text to a file, then - I agree with you! When I started with Python - which was my first programming language - I struggled a lot with understanding unicode, string encoding, and the Python types and methods that relate to it. However, the complexity is not Python's fault - it's down to the way that computers encode text, and specifically the fact that there are multiple encodings for text. Different series of bytes can represent the same series of characters, depending upon the encoding that is being used. This makes it impossible for Python to just conceal the nuts and bolts of string encoding from you and 'automatically do something sensible' like I, as a rookie, naively hoped and expected it would.

If you're going to be writing any significant amount of code involving getting and using text data from the web that may contain non-ASCII characters, I'd recommend reading around this topic thoroughly and getting a solid understanding of it, both from a general and Python-specific perspective.

Answer 2

You can try using unicode strings, like this:

jstring = u"桜の花びらたち"

See this: http://docs.python.org/tutorial/introduction.html#unicode-strings