Python how to decode unicode with hex characters

Question

I have extracted a string from web crawl script as following:

u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'

I want to decode u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91' with utf-8. With http://ddecode.com/hexdecoder/, I can see the result is '【中字】'

I tried using the following syntax but failed.

msg = u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'
result = msg.decode('utf8')

Error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-11: ordi
nal not in range(128)

May I ask how to decode the string correctly?

Thanks for help.

Answer 1

The problem with

msg = u'\xe3\x80\x90\xe4\xb8\xad\xe5\xad\x97\xe3\x80\x91'
result = msg.decode('utf8')

is that you are trying to decode Unicode. That doesn't really make sense. You can encode from Unicode to some type of encoding, or you can decode a byte string to Unicode.

When you do

msg.decode('utf8')

Python 2 sees that msg is Unicode. It knows that it can't decode Unicode so it "helpfully" assumes that you want to encode msg with the default ASCII codec so the result of that transformation can be decoded to Unicode using the UTF-8 codec. Python 3 behaves much more sensibly: that code would simply fail with

AttributeError: 'str' object has no attribute 'decode'

The technique given in kennytm's answer:

msg.encode('latin1').decode('utf-8')

works because the Unicode codepoints less than 256 correspond directly to the characters in the Latin1 encoding (aka ISO 8859-1).

Here's some Python 2 code that illustrates this:

for i in xrange(256):
    lat = chr(i)
    uni = unichr(i)
    assert lat == uni.encode('latin1')
    assert lat.decode('latin1') == uni

And here is the equivalent Python 3 code:

for i in range(256):
    lat = bytes([i])
    uni = chr(i)
    assert lat == uni.encode('latin1')
    assert lat.decode('latin1') == uni

You may find this article helpful: Pragmatic Unicode, which was written by SO veteran Ned Batchelder.

Unless you are forced to use Python 2 I strongly advise you to switch to Python 3. It will make handling Unicode far less painful.

Answer 2

1. Perhaps you should fix the crawl script instead, a Unicode string should contain u'【中字】' (u'\u3010\u4e2d\u5b57\u3011') already, instead of the raw UTF-8 bytes.

2. To convert msg to the correct encoding, first you need to turn the wrong Unicode string back to byte string (encode it as Latin-1), then decode it as UTF-8:

   >>> print msg.encode('latin1').decode('utf-8')
   【中字】