Why is this not a fixed width pattern?

Question

I'm trying to split English sentences correctly, and I came up with the unholy regex below:

(?<!\d|([A-Z]\.)|(\.[a-z]\.)|(\.\.\.)|etc\.|[Pp]rof\.|[Dd]r\.|[Mm]rs\.|[Mm]s\.|[Mm]z\.|[Mm]me\.)(?<=([\.!?])|(?<=([\.!?][\'\"])))[\s]+?(?=[\S])'

The problem is, Python keeps raising the following error:

Traceback (most recent call last):
  File "", line 1, in 
  File "sp.py", line 55, in analyze
    self.sentences = re.split(god_awful_regex, self.inputstr.strip())
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/re.py", line 165, in split
    return _compile(pattern, 0).split(string, maxsplit)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/re.py", line 243, in _compile
    raise error, v # invalid expression
sre_constants.error: look-behind requires fixed-width pattern

Why is this not a valid, fixed-width regex? I'm not using any repeat characters (* or +), just |.

---

EDIT @Anomie solved the problem - thanks a ton! Unfortunately, I cannot make the new expression balance:

(?<!(\d))(?<![A-Z]\.)(?<!\.[a-z]\.)(?<!(\.\.\.))(?<!etc\.)(?<![Pp]rof\.)(?<![Dd]r\.)(?<![Mm]rs\.)(?<![Mm]s\.)(?<![Mm]z\.)(?<![Mm]me\.)(?:(?<=[\.!?])|(?<=[\.!?][\'\"\]))[\s]+?(?=[\S])

is what I have now. The number of ('s matches the number of ('s, though:

>>> god_awful_regex = r'''(?<!(\d))(?<![A-Z]\.)(?<!\.[a-z]\.)(?<!(\.\.\.))(?<!etc\.)(?<![Pp]rof\.)(?<![Dd]r\.)(?<![Mm]rs\.)(?<![Mm]s\.)(?<![Mm]z\.)(?<![Mm]me\.)(?:(?<=[\.!?])|(?<=[\.!?][\'\"\]))[\s]+?(?=[\S])'''
>>> god_awful_regex.count('(')
17
>>> god_awful_regex.count(')')
17
>>> god_awful_regex.count('[')
13
>>> god_awful_regex.count(']')
13

Any more ideas?

Answer 1

Consider this subexpression:

(?<=([\.!?])|(?<=([\.!?][\'\"])))

The left side of the | is one character, while the right size is zero. You have the same issue in your larger negative look-behind too, it could be 1, 2, 3, 4, or 5 characters.

Logically, a negative look-behind of (?<!A|B|C) should be equivalent to a series of look-behinds (?<!A)(?<!B)(?<!C). A positive look-behind of (?<=A|B|C) should be equivalent to (?:(?<=A)|(?<=B)|(?<=C)).

Answer 2

This doesn't answer your question. However, if you want to split a text into sentences, you might want to take a look at nltk, which include beside many other things a PunktSentenceTokenizer. Here is some example tokenizer:

""" PunktSentenceTokenizer

A sentence tokenizer which uses an unsupervised algorithm to build a model
for abbreviation words, collocations, and words that start sentences; and then
uses that model to find sentence boundaries. This approach has been shown to
work well for many European languages. """

from nltk.tokenize.punkt import PunktSentenceTokenizer

tokenizer = PunktSentenceTokenizer()
print tokenizer.tokenize(__doc__)

# [' PunktSentenceTokenizer\n\nA sentence tokenizer which uses an unsupervised
# algorithm to build a model\nfor abbreviation words, collocations, and words
# that start sentences; and then\nuses that model to find sentence boundaries.',
# 'This approach has been shown to\nwork well for many European languages. ']