Python UTF-8 conversion problem

Question

In my database, I have stored some UTF-8 characters. E.g. 'α' in the "name" field

Via Django ORM, when I read this out, I get something like

>>> p.name
u'\xce\xb1'
>>> print p.name
α

I was hoping for 'α'.

After some digging, I think if I did

>>> a = 'α'
>>> a
'\xce\xb1'

So when Python is trying to display '\xce\xb1' I get alpha, but when it's trying to display u'\xce\xb1', it's double encoding?

Why did I get u'\xce\xb1' in the first place? Is there a way I can just get back '\xce\xb1'?

Thanks. My UTF-8 and unicode handling knowledge really need some help...

Answer 1

Try to put the unicode signature u before your string, e.g. u'YOUR_ALFA_CHAR' and revise your database encoding, because Django always supports UTF-8 .

Answer 2

What you seem to have is the individual bytes of a UTF-8 encoded string interpreted as unicode codepoints. You can "decode" your string out of this strange form with:

p.name = ''.join(chr(ord(x)) for x in p.name)

or perhaps

p.name = ''.join(chr(ord(x)) for x in p.name).decode('utf8')

One way to get your strings "encoded" into this form is

''.join(unichr(ord(x)) for x in '\xce\xb1')

although I have a feeling your strings actually got in this state by different components of your system disagreeing on the encoding in use.

You will probably have to fix the source of your bad "encoding" rather than just fixing the data currently in your database. And the code above might be okay to convert your bad data once, but I would advise you don't insert this code into your Django app.

Answer 3

The problem is that p.name was not correctly stored and/or read in from the database.

Unicode small alpha is U+03B1 and p.name should have printed as u'\x03b1' or if you were using a Unicode capable terminal the actual alpha symbol itself may have been printed in quotes. Note the difference between u'\xce\xb1' and u'\xceb1'. The former is a two character string and the latter in a single character string. I have no idea how the '03' byte of the UTF-8 got translated into 'CE'.