Easiest way to remove unicode representations from a string in python 3?

Question

I have a string in python 3 that has several unicode representations in it, for example:

t = 'R\\u00f3is\\u00edn'

and I want to convert t so that it has the proper representation when I print it, ie:

>>> print(t)
Róisín

However I just get the original string back. I've tried re.sub and some others, but I can't seem to find a way that will change these characters without having to iterate over each one. What would be the easiest way to do so?

Answer 1

You want to use the built-in codec unicode_escape.

If t is already a bytes (an 8-bit string), it's as simple as this:

>>> print(t.decode('unicode_escape'))
Róisín

If t has already been decoded to Unicode, you can to encode it back to a bytes and then decode it this way. If you're sure that all of your Unicode characters have been escaped, it actually doesn't matter what codec you use to do the encode. Otherwise, you could try to get your original byte string back, but it's simpler, and probably safer, to just force any non-encoded characters to get encoded, and then they'll get decoded along with the already-encoded ones:

>>> print(t.encode('unicode_escape').decode('unicode_escape')
Róisín

In case you want to know how to do this kind of thing with regular expressions in the future, note that sub lets you pass a function instead of a pattern for the repl. And you can convert any hex string into an integer by calling int(hexstring, 16), and any integer into the corresponding Unicode character with chr (note that this is the one bit that's different in Python 2—you need unichr instead). So:

>>> re.sub(r'(\\u[0-9A-Fa-f]+)', lambda matchobj: chr(int(matchobj.group(0)[2:], 16)), t)
Róisín

Or, making it a bit more clear:

>>> def unescapematch(matchobj):
...     escapesequence = matchobj.group(0)
...     digits = escapesequence[2:]
...     ordinal = int(digits, 16)
...     char = chr(ordinal)
...     return char
>>> re.sub(r'(\\u[0-9A-Fa-f]+)', unescapematch, t)
Róisín

The unicode_escape codec actually handles \U, \x, \X, octal (\066), and special-character (\n) sequences as well as just \u, and it implements the proper rules for reading only the appropriate max number of digits (4 for \u, 8 for \U, etc., so r'\\u22222' decodes to '∢2' rather than '𢈢'), and probably more things I haven't thought of. But this should give you the idea.