UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

Question

I am working google appengine python 2.5.

I am experiencing unicodedecoderror on following code because myuser name has following value

userName     = unicode(userName).encode('utf-8') # äºï¼égãwmj is value in this variable  

userName     = unicode(userName).encode('utf-8')
strData = '{\"Sid\" :1, \"Oppid\" :%s, \"Aid\" :%s, \"EC\" :\"%s\", \"Name\" :\%s"' % (enemyID, userID, userEmpCode,userName)


   params = {'deviceToken'   : oDeviceToken,
              'message'       : strMessage,
              'CertificateId' : certificateId,
              'Data'          : strData
             }


result = urlfetch.fetch(url = url,
             payload = urllib.urlencode(params),
             method  = urlfetch.POST,
             headers = {"Authorization" : authString},
             deadline = 30
             )

I am doing the following steps on username to encode it into utf-8 so that I could send it as payload.

username = unicode(username).encode(utf-8)

I believe the error occurs when I call urllib.urlencode(params)

Please guide what is going wrong.. or you can..

and what should be ultimate strategy to deal with unicode string on appengine python..

I have tried different solutions reading different threads.. but still did not work

Answer 1

You're problem seems to be that you're calling unicode(userName) without an encoding on your already-encoded string, so it "defaults to the current default string encoding", which seems to be ascii in your case.

You probably should not call unicode in any case, if you know it's a unicode value, you're fine already, if not, call .decode with the correct encoding.
If you're unsure, test using isinstance since trying to decode a unicode value will result in yet another error.

Answer 2

I had a similar issue when porting Python 3 code from Ubuntu Linux 14.04 to FreeBSD 10.3. The latter system seems to use ASCII by default instead of UTF-8 when opening files with Python 3.4.4.

Specifying encoding='utf-8' with the file open command resolved my issue:

open('filepath', encoding='utf-8')