Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Regular expression that finds and replaces non-ascii characters with Python

Tags:

python

regex

I need to change some characters that are not ASCII to '_'. For example,

Tannh‰user -> Tannh_user
  • If I use regular expression with Python, how can I do this?
  • Is there better way to do this not using RE?
like image 455
prosseek Avatar asked May 03 '10 14:05

prosseek


3 Answers

re.sub(r'[^\x00-\x7F]', '_', theString)

This will work if theString is unicode, or a string in an encoding where ASCII occupies values 0 to 0x7F (latin-1, UTF-8, etc.).

like image 144
interjay Avatar answered Oct 24 '22 21:10

interjay


To answer the question

'[\u0080-\uFFFF]'

will match any UTF-8 character not in the range of the first 128 characters

re.sub('[\u0080-\uFFFF]+', '_', x)

will replace any sequence of consecutive nonascii characters with an underscore

like image 38
Max Candocia Avatar answered Oct 24 '22 20:10

Max Candocia


Updated for Python 3:

>>> 'Tannh‰user'.encode().decode('ascii', 'replace').replace(u'\ufffd', '_')
'Tannh___user'

First we create byte string using encode() - it uses UTF-8 codec by default. If you have byte string then of course skip this encode step. Then we convert it to "normal" string using the ascii codec.

This uses the property of UTF-8 that all non-ascii characters are encoded as sequence of bytes with value >= 0x80.


Original answer – for Python 2:

How to do it using built-in str.decode method:

>>> 'Tannh‰user'.decode('ascii', 'replace').replace(u'\ufffd', '_')
u'Tannh___user'

(You get unicode string, so convert it to str if you need.)

You can also convert unicode to str, so one non-ASCII character is replaced by ASCII one. But the problem is that unicode.encode with replace translates non-ASCII characters into '?', so you don't know if the question mark was there already before; see solution from Ignacio Vazquez-Abrams.


Another way, using ord() and comparing value of each character if it fits in ASCII range (0-127) - this works for unicode strings and for str in utf-8, latin and some other encodings:

>>> s = 'Tannh‰user' # or u'Tannh‰user' in Python 2
>>> 
>>> ''.join(c if ord(c) < 128 else '_' for c in s)
'Tannh_user'
like image 6
Messa Avatar answered Oct 24 '22 21:10

Messa