How to determine if a string is escaped unicode

Question

How do you determine if a string contains escaped unicode so you know whether or not to run .decode("unicode-escape")?

For example:

test.py

# -*- coding: utf-8 -*-
str_escaped = '"A\u0026B"'
str_unicode = '"Война́ и миръ"'

arr_all_strings = [str_escaped, str_unicode]

def is_escaped_unicode(str):
    #how do I determine if this is escaped unicode?
    pass

for str in arr_all_strings:
    if is_escaped_unicode(str):
        str = str.decode("unicode-escape")
    print str

Current output:

"A\u0026B"
"Война́ и миръ"

Expected output:

"A&B"
"Война́ и миръ"

How do I define is_escaped_unicode(str) to determine if the string that's passed is actually escaped unicode?

Answer 1

You can not.

There is no way to tell if '"A\u0026B"' originally came from some text that was encoded, or if the data are just the bytes '"A\u0026B"', or if we arrived there from some other encoding.

How do ... you know whether or not to run .decode("unicode-escape")

You have to know if someone earlier has called text.encode('unicode-escape'). The bytes themselves can not tell you.

You can certainly guess, by looking for \u or \U escape sequences, or by just try/except the decoding and see what happens, but I don't recommend to go down this route.

If you encounter a bytestring in your application, and you don't already know what the encoding is, then your problem lies elsewhere and should be fixed elsewhere.

Answer 2

str_escaped = u'"A\u0026B"'
str_unicode = '"Война́ и миръ"'

arr_all_strings = [str_escaped, str_unicode]

def is_ascii(s):
    return all(ord(c) < 128 for c in s)

def is_escaped_unicode(str):
    #how do I determine if this is escaped unicode?
    if is_ascii(str): # escaped unicode is ascii
        return True
    return False

for str in arr_all_strings:
    if is_escaped_unicode(str):
        str = str.decode("unicode-escape")
    print str

The following code will work for your case.

Explain:

• All string in str_escaped is in Ascii range.

• Char in str_unicode do not contain in Ascii range.

Answer 3

Here's a crude way to do it. Try decoding as unicode-escape, and if that succeeds the resulting string will be shorter than the original string.

str_escaped = '"A\u0026B"'
str_unicode = '"Война́ и миръ"'
arr_all_strings = [str_escaped, str_unicode]

def decoder(s):
    y = s.decode('unicode-escape')
    return y if len(y) < len(s) else s.decode('utf8')

for s in arr_all_strings:
    print s, decoder(s)

output

"A\u0026B" "A&B"
"Война и миръ" "Война и миръ"

---

But seriously, you'll save yourself a lot of pain if you can migrate to Python 3. And if you can't immediately migrate to Python 3, you may find this article helpful: Pragmatic Unicode, which was written by SO veteran Ned Batchelder.