How to output a utf-8 string list as it is in python?

Question

Well, character encoding and decoding sometimes frustrates me a lot.

So we know u'\u4f60\u597d' is the utf-8 encoding of 你好,

>>> print hellolist
[u'\u4f60\u597d']
>>> print hellolist[0]
你好

Now what I really want to get from the output or write to a file is [u'你好'], but it's [u'\u4f60\u597d'] all the time, so how do you do it?

Answer 1

When you print (or write to a file) a list it internally calls the str() method of the list , but list internally calls repr() on its elements. repr() returns the ugly unicode representation that you are seeing .

Example of repr -

>>> h = u'\u4f60\u597d'
>>> print h
\u4f60\u597d
>>> print repr(h)
u'\u4f60\u597d'

You would need to manually take the elements of the list and print them for them to print correctly.

Example -

>>> h1 = [h,u'\u4f77\u587f']
>>> print u'[' + u','.join([u"'" + unicode(i) + u"'" for i in h1]) + u']'

For lists containing sublists that may have unicode characters, you would need a recursive function , example -

>>> h1 = [h,(u'\u4f77\u587f',)]
>>> def listprinter(l):
...     if isinstance(l, list):
...             return u'[' + u','.join([listprinter(i) for i in l]) + u']'
...     elif isinstance(l, tuple):
...             return u'(' + u','.join([listprinter(i) for i in l]) + u')'
...     elif isinstance(l, (str, unicode)):
...             return u"'" + unicode(l) + u"'"
... 
>>> 
>>> 
>>> print listprinter(h1)

To save them to file, use the same list comprehension or recursive function. Example -

with open('<filename>','w') as f:
    f.write(listprinter(l))

Answer 2

You are misunderstanding.

u'' in python is not utf-8, it is simply Unicode (except on Windows in Python <= 3.2, where it is utf-16 instead).

utf-8 is an encoding of Unicode, which is necessarily a sequence of bytes.

Additionally, u'你' and u'\u4f60' are exactly the same thing. It's simply that in Python2 the repr of high characters uses escapes instead of raw values.

Since Python2 is heading for EOL very soon now, you should start to think seriously about switching to Python3. It is a lot easier to keep track of all this in Python3 since there's only one string type and it's much more clear when you .encode and .decode.