I need to change some characters that are not ASCII to '_'. For example,
Tannh‰user -> Tannh_user
re.sub(r'[^\x00-\x7F]', '_', theString)
This will work if theString is unicode, or a string in an encoding where ASCII occupies values 0 to 0x7F (latin-1, UTF-8, etc.).
To answer the question
'[\u0080-\uFFFF]'
will match any UTF-8 character not in the range of the first 128 characters
re.sub('[\u0080-\uFFFF]+', '_', x)
will replace any sequence of consecutive nonascii characters with an underscore
Updated for Python 3:
>>> 'Tannh‰user'.encode().decode('ascii', 'replace').replace(u'\ufffd', '_')
'Tannh___user'
First we create byte string using encode()
- it uses UTF-8 codec by default. If you have byte string then of course skip this encode step.
Then we convert it to "normal" string using the ascii codec.
This uses the property of UTF-8 that all non-ascii characters are encoded as sequence of bytes with value >= 0x80.
Original answer – for Python 2:
How to do it using built-in str.decode
method:
>>> 'Tannh‰user'.decode('ascii', 'replace').replace(u'\ufffd', '_')
u'Tannh___user'
(You get unicode
string, so convert it to str
if you need.)
You can also convert unicode
to str
, so one non-ASCII character is replaced by ASCII one. But the problem is that unicode.encode
with replace
translates non-ASCII characters into '?'
, so you don't know if the question mark was there already before; see solution from Ignacio Vazquez-Abrams.
Another way, using ord()
and comparing value of each character if it fits in ASCII range (0-127) - this works for unicode
strings and for str
in utf-8, latin and some other encodings:
>>> s = 'Tannh‰user' # or u'Tannh‰user' in Python 2
>>>
>>> ''.join(c if ord(c) < 128 else '_' for c in s)
'Tannh_user'
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With