regex to parse string with escaped characters

Question

I am reading information out of a formatted string. The format looks like this:

"foo:bar:beer:123::lol"

Everything between the ":" is data I want to extract with regex. If a : is followed by another : (like "::") the data for this has to be "" (an empty string).

Currently I am parsing it with this regex:

(.*?)(:|$)

Now it came to my mind that ":" may exist within the data, as well. So it has to be escaped. Example:

"foo:bar:beer:\::1337"

How can I change my regular expression so that it matches the "\:" as data, too?

Edit: I am using JavaScript as programming language. It seems to have some limitations regarding complex regulat expressions. The solution should work in JavaScript, as well.

Thanks, McFarlane

Answer 1

var myregexp = /((?:\\.|[^\\:])*)(?::|$)/g;
var match = myregexp.exec(subject);
while (match != null) {
    for (var i = 0; i < match.length; i++) {
        // Add match[1] to the list of matches
    }
    match = myregexp.exec(subject);
}

Input: "foo:bar:beer:\\:::1337"

Output: ["foo", "bar", "beer", "\\:", "", "1337", ""]

You'll always get an empty string as the last match. This is unavoidable given the requirement that you also want empty strings to match between delimiters (and the lack of lookbehind assertions in JavaScript).

Explanation:

(          # Match and capture:
 (?:       # Either match...
  \\.      # an escaped character
 |         # or
  [^\\:]   # any character except backslash or colon
 )*        # zero or more times
)          # End of capturing group
(?::|$)    # Match (but don't capture) a colon or end-of-string

Answer 2

Here's a solution:

function tokenize(str) {
  var reg = /((\\.|[^\\:])*)/g;
  var array = [];
  while(reg.lastIndex < str.length) {
    match = reg.exec(str);
    array.push(match[0].replace(/\\(\\|:)/g, "$1"));
    reg.lastIndex++;
  }
  return array;
}

It splits a string into token depending on the : character.

• But you can escape the : character with \ if you want it to be part of a token.
• you can escape the \ with \ if you want it to be part of a token
• any other \ won't be interpreted. (ie: \a remains \a)
• So you can put any data in your tokens provided that data is correctly formatted before hand.

Here is an example with the string \a:b:\n::\\:\::x, which should give these token: \a, b, \n, <empty string>, \, :, x.

>>> tokenize("\\a:b:\\n::\\\\:\\::x");
["\a", "b", "\n", "", "\", ":", "x"]

In an attempt to be clearer: the string put into the tokenizer will be interpreted, it has 2 special character: \ and :

• \ will only have a special meaning only if followed by \ or :, and will effectively "escape" these character: meaning that they will loose their special meaning for tokenizer, and they'll be considered as any normal character (and thus will be part of tokens).
• : is the marker separating 2 tokens.

I realize the OP didn't ask for slash escaping, but other viewers could need a complete parsing library allowing any character in data.

Answer 3

Use a negative lookbehind assertion.

(.*?)((?<!\\):|$)

This will only match : if it's not preceded by \.