Regular expression in python: removing square brackets and parts of the phrase inside of the brackets

Question

I have a wikipedia dump and struggling with finding appropriate regex patter to remove the double square brackets in the expression. Here is the example of the expressions:

line = 'is the combination of the code names for Herbicide Orange (HO) and Agent LNX, one of the [[herbicide]]s and [[defoliant]]s used by the [[United States armed forces|U.S. military]] as part of its [[herbicidal warfare]] program, [[Operation Ranch Hand]], during the [[Vietnam War]] from 1961 to 1971.'

I am looking to remove all of the square brackets with the following conditions:

• if there is no vertical separator within square bracket, remove the brackets.

  Example : [[herbicide]]s becomes herbicides.

• if there is a vertical separator within the bracket, remove the bracket and only use the phrase after the separator.

  Example : [[United States armed forces|U.S. military]] becomes U.S. military.

I tried using re.match and re.search but was not able to arrive to the desired output.

Thank you for your help!

Answer 1

What you need is re.sub. Note that both square brackets and pipes are meta-characters so they need to be escaped.

re.sub(r'\[\[(?:[^\]|]*\|)?([^\]|]*)\]\]', r'\1', line)

The \1 in the replacement string refers to what was matched inside the parentheses, that do not start with ?: (i.e. in any case the text you want to have).

There are two caveats. This allows for only a single pipe between the opening and closing brackets. If there are more than one you would need to specify whether you want everything after the first or everything after the last one. The other caveat is that single ] between opening and closing brackets are not allowed. If that is a problem, there would still be a regex solution but it would be considerably more complicated.

For a full explanation of the pattern:

\[\[        # match two literal [
(?:         # start optional non-capturing subpattern for pre-| text
   [^\]|]   # this looks a bit confusing but it is a negated character class
            # allowing any character except for ] and |
   *        # zero or more of those
   \|       # a literal |
)?          # end of subpattern; make it optional
(           # start of capturing group 1 - the text you want to keep
    [^\]|]* # the same character class as above
)           # end of capturing group
\]\]        # match two literal ]

Answer 2

>>> import re
>>> re.sub(r'\[\[(?:[^|\]]*\|)?([^\]]*)]]', r'\1', line)
'is the combination of the code names for Herbicide Orange (HO) and Agent LNX, one of the herbicides and defoliants used by the U.S. military as part of its herbicidal warfare program, Operation Ranch Hand, during the Vietnam War from 1961 to 1971.'

Explanation:

\[\[       # match two opening square brackets
(?:        # start optional non-capturing group
   [^|\]]*   # match any number of characters that are not '|' or ']'
   \|        # match a '|'
)?         # end optional non-capturing group
(          # start capture group 1
   [^\]]*    # match any number of characters that are not ']'
)          # end capture group 1
]]         # match two closing square brackets

By replacing matches of the above regex with the contents of capture group 1, you will get the contents of the square brackets, but only what is after the separator if it is present.