Python re.sub(): how to substitute all 'u' or 'U's with 'you'

Question

I am doing some text normalization using python and regular expressions. I would like to substitute all 'u'or 'U's with 'you'. Here is what I have done so far:

import re text = 'how are u? umberella u! u. U. U@ U# u ' print re.sub (' [u|U][s,.,?,!,W,#,@ (^a-zA-Z)]', ' you ', text) 

The output I get is:

how are you  you berella you  you  you  you  you  you 

As you can see the problem is that 'umberella' is changed to 'berella'. Also I want to keep the character that appears after a 'u'. For example I want 'u!' to be changed to 'you!'. Can anyone tell me what I am doing wrong and what is the best way to write the regular expression?

Answer 1

Firstly, why doesn't your solution work. You mix up a lot of concepts. Mostly character class with other ones. In the first character class you use | which stems from alternation. In character classes you don't need the pipe. Just list all characters (and character ranges) you want:

[Uu] 

Or simply write u if you use the case-insensitive modifier. If you write a pipe there, the character class will actually match pipes in your subject string.

Now in the second character class you use the comma to separate your characters for some odd reason. That does also nothing but include commas into the matchable characters. s and W are probably supposed to be the built-in character classes. Then escape them! Otherwise they will just match literal s and literal W. But then \W already includes everything else you listed there, so a \W alone (without square brackets) would have been enough. And the last part (^a-zA-Z) also doesn't work, because it will simply include ^, (, ) and all letters into the character class. The negation syntax only works for entire character classes like [^a-zA-Z].

What you actually want is to assert that there is no letter in front or after your u. You can use lookarounds for that. The advantage is that they won't be included in the match and thus won't be removed:

r'(?<![a-zA-Z])[uU](?![a-zA-Z])' 

Note that I used a raw string. Is generally good practice for regular expressions, to avoid problems with escape sequences.

These are negative lookarounds that make sure that there is no letter character before or after your u. This is an important difference to asserting that there is a non-letter character around (which is similar to what you did), because the latter approach won't work at the beginning or end of the string.

Of course, you can remove the spaces around you from the replacement string.

If you don't want to replace u that are next to digits, you can easily include the digits into the character classes:

r'(?<![a-zA-Z0-9])[uU](?![a-zA-Z0-9])' 

And if for some reason an adjacent underscore would also disqualify your u for replacement, you could include that as well. But then the character class coincides with the built-in \w:

r'(?<!\w)[uU](?!\w)' 

Which is, in this case, equivalent to EarlGray's r'\b[uU]\b'.

As mentioned above you can shorten all of these, by using the case-insensitive modifier. Taking the first expression as an example:

re.sub(r'(?<![a-z])u(?![a-z])', 'you', text, flags=re.I) 

or

re.sub(r'(?<![a-z])u(?![a-z])', 'you', text, flags=re.IGNORECASE) 

depending on your preference.

I suggest that you do some reading through the tutorial I linked several times in this answer. The explanations are very comprehensive and should give you a good headstart on regular expressions, which you will probably encounter again sooner or later.

Answer 2

Use a special character \b, which matches empty string at the beginning or at the end of a word:

print re.sub(r'\b[uU]\b', 'you', text) 

spaces are not a reliable solution because there are also plenty of other punctuation marks, so an abstract character \b was invented to indicate a word's beginning or end.