Regular Expression to escape HTML ampersands while respecting CDATA

Question

I've written a content management system that uses a server-side regular expression to escape ampersands in the page response just prior to it being sent to the client's browser. The regular expression is mindful of ampersands that have already been escaped or are part of an HTML entity. For example, the following:

a & b, c & d, © 2009

gets changed to this:

a & b, c & d, © 2009

(Only the first & is modified.) Here is the regular expression, which was taken and modified from a Rails helper:

html.gsub(/&(?!([a-zA-Z][a-zA-Z0-9]*|(#\d+));)/) { |special| ERB::Util::HTML_ESCAPE[special] }

While this works great, it does have a problem. The regular expression is not aware of any <![CDATA[ or ]]> that might be surrounding the unescaped ampersands. This is necessary for embedded JavaScript to remain untouched. For example, this:

<script type="text/javascript">
  // <![CDATA[
  if (a && b) doSomething();
  // ]]>
</script>

is unfortunately rendered as this:

<script type="text/javascript">
  // <![CDATA[
  if (a &amp;&amp; b) doSomething();
  // ]]>
</script>

which of course the JavaScript engines don't understand.

My question is this: Is there a way to modify the regular expression to do exactly as it is doing now with the exception that it leaves text inside a CDATA section untouched?

Since the regular expression is not so simple to begin with, this question might be easier to answer: Is it possible to write a regular expression that will change all letters into a period except those letters between a '<' and a '>'? For example, one that would change "some <words> are < safe! >" into ".... <words> ... < safe! >"?

Answer 1

You asked for it! :D

/&(?!(?:[a-zA-Z][a-zA-Z0-9]*|#\d+);)
 (?!(?>(?:(?!<!\[CDATA\[|\]\]>).)*)\]\]>)/xm

The first line is your original regex. The lookahead matches if there's a CDATA closing sequence ( ]]> ) up ahead, unless there's an opening sequence ( <!CDATA[ ) between here and there. Assuming the document is minimally well formed, that should mean the current position is inside a CDATA section.

Oops, I had that backward: by using positive lookahead I was matching "naked" ampersands only within CDATA sections. I changed it to a negative lookahead, so now it works right.

By the way, this regex works in RegexBuddy in Ruby mode, but not at the rubular site. I suspect Rubular uses an older version of Ruby with less-powerful regex support; can anyone confirm that? (As you may have guessed, I'm not a Ruby programmer.)

EDIT: The problem at Rubular was that I used 's' as a modifier (to mean dot-matches-everything), but Ruby uses 'm' for that.

Answer 2

Don't use regular expressions for this. It is a terrible, terrible idea. Instead, simply HTML encode anything that you're outputting that might have a character in it. Like this:

require 'cgi'
print CGI.escape("All of this is HTML encoded!")

Answer 3

That worked! At Rubular I had to change the options from /xs to /m (and I removed the whitespace that separates the two parts of the regex as you showed it above).

You can see this regular expression in action along with a sample string at http://www.rubular.com/regexes/5855.

In case that Rubular permalink isn't really permanent, here is what I entered for the regular expression:

/&(?!(?:[a-zA-Z][a-zA-Z0-9]*|#\d+);)(?!(?>(?:(?!<!\[CDATA\[|\]\]>).)*)\]\]>)/m

And here is the test string:

<p>a & b</p>
<p>c &amp; d</p>
<script type="text/javascript">
  // <![CDATA[
  if (a && b) doSomething('a & b &amp; c');
  // ]]>
</script>
<p>a & b</p>
<p>c &amp; d</p>

Only two ampersands match -- the a & b at the top and the a & b at the bottom. Ampersands already escaped as & and all ampersands (escaped or not) between <![CDATA[ and ]]> are left alone.

So, my final code is now this:

html.gsub(/&(?!(?:[a-zA-Z][a-zA-Z0-9]*|#\d+);)(?!(?>(?:(?!<!\[CDATA\[|\]\]>).)*)\]\]>)/m, '&amp;')

Thank you very much Alan. This is exactly what I needed.