finding regular expression literals in a string of javascript code

Question

I am doing a sort of crude parsing of javascript code, with javascript. I'll spare the details of why I need to do this, but suffice to say that I don't want to integrate a huge chunk of library code, as it is unnecessary for my purposes and it is important that I keep this very lightweight and relatively simple. So please don't suggest I use JsLint or anything like that. If the answer is more code than you can paste into your answer, it's probably more than I want.

My code currently is able to do a good job of detecting quoted sections and comments, and then matching braces, brackets and parens (making sure not to be confused by the quotes and comments, or escapes within quotes, of course). This is all I need it to do, and it does it well...with one exception:

It can be confused by regular expression literals. So I'm hoping for some help with detecting regular expression literals in a string of javascript, so I can handle them appropriately.

Something like this:

function getRegExpLiterals (stringOfJavascriptCode) {
  var output = [];
  // todo!
  return output;
}

var jsString =  "var regexp1 = /abcd/g, regexp1 = /efg/;"
console.log (getRegExpLiterals (jsString));

// should print:
// [{startIndex: 13, length: 7}, {startIndex: 32, length: 5}]

Answer 1

es5-lexer is a JS lexer that uses a very accurate heuristic to distinguish regular expressions in JS code from division expressions, and also provides a token level transformation that you can use to make sure that the resulting program will be interpreted the same way by a full JS parser as by the lexer.

The bit that determines whether a / starts a regular expression is in guess_is_regexp.js and the tests start at scanner_test.js line 401

var REGEXP_PRECEDER_TOKEN_RE = new RegExp(
  "^(?:"  // Match the whole tokens below
    + "break"
    + "|case"
    + "|continue"
    + "|delete"
    + "|do"
    + "|else"
    + "|finally"
    + "|in"
    + "|instanceof"
    + "|return"
    + "|throw"
    + "|try"
    + "|typeof"
    + "|void"
    // Binary operators which cannot be followed by a division operator.
    + "|[+]"  // Match + but not ++.  += is handled below.
    + "|-"    // Match - but not --.  -= is handled below.
    + "|[.]"    // Match . but not a number with a trailing decimal.
    + "|[/]"  // Match /, but not a regexp.  /= is handled below.
    + "|,"    // Second binary operand cannot start a division.
    + "|[*]"  // Ditto binary operand.
  + ")$"
  // Or match a token that ends with one of the characters below to match
  // a variety of punctuation tokens.
  // Some of the single char tokens could go above, but putting them below
  // allows closure-compiler's regex optimizer to do a better job.
  // The right column explains why the terminal character to the left can only
  // precede a regexp.
  + "|["
    + "!"  // !           prefix operator operand cannot start with a division
    + "%"  // %           second binary operand cannot start with a division
    + "&"  // &, &&       ditto binary operand
    + "("  // (           expression cannot start with a division
    + ":"  // :           property value, labelled statement, and operand of ?:
           //             cannot start with a division
    + ";"  // ;           statement & for condition cannot start with division
    + "<"  // <, <<, <<   ditto binary operand
    // !=, !==, %=, &&=, &=, *=, +=, -=, /=, <<=, <=, =, ==, ===, >=, >>=, >>>=,
    // ^=, |=, ||=
    // All are binary operands (assignment ops or comparisons) whose right
    // operand cannot start with a division operator
    + "="
    + ">"  // >, >>, >>>  ditto binary operand
    + "?"  // ?           expression in ?: cannot start with a division operator
    + "["  // [           first array value & key expression cannot start with
           //             a division
    + "^"  // ^           ditto binary operand
    + "{"  // {           statement in block and object property key cannot start
           //             with a division
    + "|"  // |, ||       ditto binary operand
    + "}"  // }           PROBLEMATIC: could be an object literal divided or
           //             a block.  More likely to be start of a statement after
           //             a block which cannot start with a /.
    + "~"  // ~           ditto binary operand
  + "]$"
  // The exclusion of ++ and -- from the above is also problematic.
  // Both are prefix and postfix operators.
  // Given that there is rarely a good reason to increment a regular expression
  // and good reason to have a post-increment operator as the left operand of
  // a division (x++ / y) this pattern treats ++ and -- as division preceders.
  );