UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)

Question

when i run my code i get this error:

UserId = "{}".format(source[1]) UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)

My code is:

def view_menu(type, source, parameters):
    ADMINFILE = 'static/users.txt'
    fp = open(ADMINFILE, 'r')
    users = ast.literal_eval(fp.read())
    if not parameters:
        if not source[1] in users:
            UserId = "{}".format(source[1])
            users.append(UserId)
            write_file(ADMINFILE,str(users))
            fp.close()
            reply(type, source, u"test")
        else:
            reply(type, source, u"test")

register_command_handler(view_menu, 'test', ['info','muc','all'], 0, '')

Please how i can solve this problem.

Thank you

Answer 1

The problem is that "{}" is non-Unicode str, and you're trying to format a unicode into it. Python 2.x handles that by automatically encoding the unicode with sys.getdefaultencoding(), which is usually 'ascii', but you have some non-ASCII characters.

There are two ways to solve this:

1. Explicitly encode that unicode in the appropriate character set. For example, if it's UTF-8, do "{}".format(source[1].encode('utf-8')).

2. Use a unicode format string: u"{}".format(source[1]). You may still need to encode that UserId later; I have no idea how your write_file function works. But it's generally better to keep everything Unicode as long as possible, only encoding and decoding at the very edges, than to try to mix and match the two.

All that being said, this line of code is useless. "{}".format(foo) converts foo to a str, and then formats it into the exact same str. Why?

Answer 2

Take these functions here when handling strings of unknown encoding:

You want to work with the text?

def read_unicode(text, charset='utf-8'):
    if isinstance(text, basestring):
        if not isinstance(text, unicode):
            text = unicode(obj, charset)
    return text

You want to store the text, for example in a database, use this:

def write_unicode(text, charset='utf-8'):
    return text.encode(charset)