UnicodeEncodeError: 'ascii' codec can't encode character u'\xe7' in position 17710: ordinal not in range(128)

Question

I'm trying to print a string from an archived web crawl, but when I do I get this error:

print page['html']
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe7' in position 17710: ordinal not in range(128)

When I try print unicode(page['html']) I get:

print unicode(page['html'],errors='ignore')
TypeError: decoding Unicode is not supported

Any idea how I can properly code this string, or at least get it to print? Thanks.

Answer 1

You need to encode the unicode you saved to display it, not decode it -- unicode is the unencoded form. You should always specify an encoding, so that your code will be portable. The "usual" pick is utf-8:

print page['html'].encode('utf-8')

If you don't specify an encoding, whether or not it works will depend on what you're printing to -- your editor, OS, terminal program, etc.