Strip unicode character modifiers

Question

What is the simplest way to strip the character modifiers from a unicode string in Python?

For example:

A͋͠r͍̞̫̜͌ͦ̈́͐ͅt̼̭͞h́u̡̙̞̘̙̬͖͓rͬͣ̐ͮͥͨ̀͏̣ should become Arthur

I tried the docs but I couldn't find anything that does this.

Answer 1

Try this

import unicodedata
a = u"STRING GOES HERE" # using an actual string would break stackoverflow's code formatting.
u"".join( x for x in a if not unicodedata.category(x).startswith("M") )

This will remove all characters classified as marks, which is what I think you want. In general, you can get the category of a character with unicodedata.category.

Answer 2

You could also use r'\p{M}' that is supported by regex module:

import regex

def remove_marks(text):
    return regex.sub(ur"\p{M}+", "", text)

Example:

>>> print s
A͋͠r͍̞̫̜t̼̭͞h́u̡̙̞̘rͬͣ̐ͮ
>>> def remove_marks(text):
...     return regex.sub(ur"\p{M}+", "", text)
...     
... 
>>> print remove_marks(s)
Arthur

Depending on your use-case a whitelist approach might be better e.g., to limit the input only to ascii characters:

>>> s.encode('ascii', 'ignore').decode('ascii')
u'Arthur'

The result might depend on Unicode normalization used in the text.