REXML :: RuntimeError (entity expansion has grown too large)

Question

After upgrading to Ruby-1.9.3-p392 today, REXML throws a Runtime Error when attempting to retrieve an XML response over a certain size - everything works fine and no error is thrown when receiving under 25 XML records, but once a certain XML response length threshold is reached, I get this error:

Error occurred while parsing request parameters.
Contents:

RuntimeError (entity expansion has grown too large):
  /.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/rexml/text.rb:387:in `block in unnormalize'

I realize this was changed in the most recent Ruby version: http://www.ruby-lang.org/en/news/2013/02/22/rexml-dos-2013-02-22/

As a quick fix, I've changed the size of REXML::Document.entity_expansion_text_limit to a larger number and the error goes away.

Is there a less risky solution?

Answer 1

This issue is generated when you send too much content as XML response.

To fix this issue : You need to restrict the data(< 10k) in the individual node (Instead of sending the whole data, show truncated data and provide a seperate link to view full content)

The error is being raised from the below file : ruby-2.1.2/lib/ruby/2.1.0/rexml/text.rb

# Unescapes all possible entities
def Text::unnormalize( string, doctype=nil, filter=nil, illegal=nil )
  sum = 0
  string.gsub( /\r\n?/, "\n" ).gsub( REFERENCE ) {
    s = Text.expand($&, doctype, filter)
    if sum + s.bytesize > Security.entity_expansion_text_limit
      raise "entity expansion has grown too large"
    else
      sum += s.bytesize
    end
    s
  }
end

The limit ruby-2.1.2/lib/ruby/2.1.0/rexml/text.rb defaults to 10240 which means 10k data per node.

REXML already defaults to only allow 10000 entity substitutions per document, so the maximum amount of text that can be generated by entity substitution will be around 98 megabytes. (Refer https://www.ruby-lang.org/en/news/2013/02/22/rexml-dos-2013-02-22/ )