Correctly parsing string literals with python's re module

Question

I'm trying to add some light markdown support for a javascript preprocessor which I'm writing in Python.

For the most part it's working, but sometimes the regex I'm using is acting a little odd, and I think it's got something to do with raw-strings and escape sequences.

The regex is: (?<!\\)\"[^\"]+\"

Yes, I am aware that it only matches strings beginning with a " character. However, this project is born out of curiosity more than anything, so I can live with it for now.

To break it down:

(?<\\)\"    # The group should begin with a quotation mark that is not escaped
[^\"]+      # and match any number of at least one character that is not a quotation mark (this is the biggest problem, I know)
\"          # and end at the first quotation mark it finds

That being said, I (obviously) start hitting problems with things like this:

"This is a string with an \"escaped quote\" inside it"

I'm not really sure how to say "Everything but a quotation mark, unless that mark is escaped". I tried:

([^\"]|\\\")+     # a group of anything but a quote or an escaped quote

, but that lead to very strange results.

I'm fully prepared to hear that I'm going about this all wrong. For the sake of simplicity, let's say that this regex will always start and end with double quotes (") to avoid adding another element in the mix. I really want to understand what I have so far.

Thanks for any assistance.

EDIT

As a test for the regex, I'm trying to find all string literals in the minified jQuery script with the following code (using the unutbu's pattern below):

STRLIT = r'''(?x)   # verbose mode
    (?<!\\)    # not preceded by a backslash
    "          # a literal double-quote
    .*?        # non-greedy 1-or-more characters
    (?<!\\)    # not preceded by a backslash
    "          # a literal double-quote
    ''' 
f = open("jquery.min.js","r")
jq = f.read()
f.close()
literals = re.findall(STRLIT,jq)

The answer below fixes almost all issues. The ones that do arise are within jquery's own regular expressions, which is a very edge case. The solution no longer misidentifies valid javascript as markdown links, which was really the goal.

Answer 1

I think I first saw this idea in... Jinja2's source code? Later transplanted it to Mako.

r'''(\"\"\"|\'\'\'|\"|\')((?<!\\)\\\1|.)*?\1'''

Which does the following:

• (\"\"\"|\'\'\'|\"|\') matches a Python opening quote, because this happens to be taken from code for parsing Python. You probably don't need all those quote types.
• ((?<!\\)\\\1|.) matches: EITHER a matching quote that was escaped ONLY ONCE, OR any other character. So \\" will still be recognized as the end of the string.
• *? non-greedily matches as many of those as possible.
• And \1 is just the closing quote.

Alas, \\\" will still incorrectly be detected as the end of the string. (The template engines only use this to check if there is a string, not to extract it.) This is a problem very poorly suited for regular expressions; short of doing insane things in Perl, where you can embed real code inside a regex, I'm not sure it's possible even with PCRE. Though I'd love to be proven wrong. :) The killer is that (?<!...) has to be constant-length, but you want to check that there's any even number of backslashes before the closing quote.

If you want to get this correct, and not just mostly-correct, you might have to use a real parser. Have a look at parsley, pyparsing, or any of these tools.

edit: By the way, there's no need to check that the opening quote doesn't have a backslash before it. That's not valid syntax outside a string in JS (or Python).