Python - Decode utf-8 list in lists (decode entire list objects)

Question

Assume that i have a list which has more than one list for example:

l = [['a'],['a','b'],['c'],['d',['a','b'],'f']]

with this:

l = [x.decode('UTF8') for x in l]

probably i will get error: list object has no attribute 'decode'

("l" list created from tokenized text which has its every words made list object. Tried many solution for overcome decode struggle but still cant print non-ascii characters)

with open(path, "r") as myfile:
    text=myfile.read()

text = word_tokenize(text)

d = [[item] if not isinstance(item, list) else item for item in text]

arr = sum(([[x[0] for x in g]] if k else list(g)
     for k, g in groupby(d, key=lambda x: x[0][0].isupper())),
    [])

arr = [x.decode('UTF8') for x in arr]

INPUT (my text file):

Çanakkale çok güzel bir şehirdir. Çok beğendik.

OUTPUT :

[[u'\xc7anakkale'], [u'\xe7ok'], [u'g\xfczel'], [u'bir'], [u'\u015fehirdir'], [u'.']. [u'\xe7ok'], [u'be\u011fendik'], [u'.']]

my desired output is list but exactly like my input format.

Answer 1

Firstly, the problem you think you have is that you're printing the whole list (you haven't included that part in your question so I've had to guess) - Python is printing the safe representation of the data. For you this means it's indicating you have Unicode strings (hence the u'') and it's showing the Unicode point hex value of the non-ASCII characters.

If you were to print an individual part of the list then you'd get what you expect.

I.e.

>>> print arr[0][0]
Çanakkale

If you want to print all the values the you'll need a for loop:

for x in arr:
    for y in x:
        print y

You're also introducing unnecessary complexity by manually decoding the data deep in your code - instead you should decode the data on input.

It appears that you're using Python 2.x (by the u'' prefixes), so use the io module to decode the text data as you read it:

import io
with io.open(path, "r", encoding="utf-8") as myfile:
    text=myfile.read()

Now you can remove the arr = [x.decode('UTF8') for x in arr] line.