Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to decode a unicode string Python [duplicate]

What is the best way to decode an encoded string that looks like: u'u\xf1somestring' ?

Background: I have a list that contains random values (strings and integers), I'm trying to convert every item in the list to a string then process each of them.

Turns out some of the items are of the format: u'u\xf1somestring' When I tried converting to a string, I get the error: UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 1: ordinal not in range(128)

I have tried

item = u'u\xf1somestring'
decoded_value = item.decode('utf-8', 'ignore')

However, I keep getting the same error.

I have read up about unicode characters and tried a number of suggestions from SO but none have worked so far. Am I missing something here?

like image 728
mfalade Avatar asked Jan 29 '16 11:01

mfalade


2 Answers

You need to call encode function and not decode function, as item is already decoded.

Like this:

decoded_value = item.encode('utf-8')
like image 62
Sameer Mirji Avatar answered Oct 17 '22 03:10

Sameer Mirji


That string already is decoded (it's a Unicode object). You need to encode it if you want to store it in a file (or send it to a dumb terminal etc.).

Generally, when working with Unicode, you should (in Python 2) decode all your strings early in the workflow (which you already seem to have done; many libraries that handle internet traffic will already do that for you), then do all your work on Unicode objects, and then at the very end, when writing them back, encode them to whatever encoding you're using.

like image 30
Tim Pietzcker Avatar answered Oct 17 '22 04:10

Tim Pietzcker