I wrote a piece of Rack Middleware to automatically unzip compressed request bodies. The code seems to be working just fine, but when I plug it into my rails app, I get a failure "Invalid JSON" from ActionController::ParamsParser.
As a debugging mechanism, I'm writing both the zipped content, and the unzipped content to a file (to make sure that code is working properly) and I do receive my original JSON document (before the client zips it up).
The data I'm posting is JSON data, and the unzipped content is detected as valid JSON from http://jsonlint.com.
Any ideas what I'm doing wrong?
class CompressedRequests
def initialize(app)
@app = app
end
def call(env)
input = env['rack.input'].read
#output the zipped data we received
File.open('/Users/ben/Desktop/data.gz', 'w+') do |f|
f.write input
end
if env['REQUEST_METHOD'] =~ /(POST|PUT)/
if env.keys.include? 'HTTP_CONTENT_ENCODING'
new_input = decode(input, env['HTTP_CONTENT_ENCODING'])
env['rack.input'] = StringIO.new(new_input)
#output our decoded data (for debugging)
File.open('/Users/ben/Desktop/data.txt', 'w+') do |f|
f.write env['rack.input'].read
end
env.delete('HTTP_CONTENT_ENCODING')
end
end
env['rack.input'].rewind
status, headers, response = @app.call(env)
return [status, headers, response]
end
def decode(input, content_encoding)
if content_encoding == 'gzip'
Zlib::GzipReader.new(input).read
elsif content_encoding == 'deflate'
Zlib::Inflate.new.inflate new input
else
input
end
end
end
Here is the error that I'm getting from the console:
Contents::"2010-05-17T12:46:30Z","background":false},{"longitude":-95.38620785000001,"latitude":29.62815358333334,"estimated_speed":14.04305,"timestamp":"2010-05-17T12:46:36Z","background":false},{"longitude":-95.3862767,"latitude":29.62926725,"estimated_speed":39.87791,"timestamp":"2010-05-17T12:46:42Z","background":false},{"longitude":-95.38655023333334,"latitude":29.63051011666666,"estimated_speed":46.09239,"timestamp":"2010-05-17T12:46:49Z","background":false},{"longitude":-95.38676226666666,"latitude":29.63158775,"estimated_speed":47.34936,"timestamp":"2010-05-17T12:46:55Z","background":false},{"longitude":-95.38675346666666,"latitude":29.63219841666666,"estimated_speed":22.54016,"timestamp":"2010-05-17T12:47:03Z","background":false},{"longitude":-95.38675491666666,"latitude":29.63265714999999,"estimated_speed":14.03642,"timestamp":"2010-05-17T12:47:10Z","background":false},{"longitude":-95.38677551666666,"latitude":29.63358661666667,"estimated_speed":29.29489,"timestamp":"2010-05-17T12:47:17Z","background":false},{"longitude":-95.38679026666662,"latitude":29.63466445,"estimated_speed":38.34926,"timestamp":"2010-05-17T12:47:24Z","background":false},{"longitude":-95.38681656666668,"latitude":29.63590941666666,"estimated_speed":44.82093,"timestamp":"2010-05-17T12:47:31Z","background":false},{"longitude":-95.38683366666667,"latitude":29.63679638333334,"estimated_speed":40.21729,"timestamp":"2010-05-17T12:47:37Z","background":false},{"longitude":-95.38685133333333,"latitude":29.63815714999999,"estimated_speed":44.86543,"timestamp":"2010-05-17T12:47:44Z","background":false},{"longitude":-95.3868655
/!\ FAILSAFE /!\ Mon Oct 18 18:18:43 -0500 2010
Status: 500 Internal Server Error
Invalid JSON string
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.5/lib/active_support/json/backends/yaml.rb:14:in `decode'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.5/lib/active_support/json/decoding.rb:11:in `__send__'
/Library/Ruby/Gems/1.8/gems/activesupport-2.3.5/lib/active_support/json/decoding.rb:11:in `decode'
/Library/Ruby/Gems/1.8/gems/actionpack-2.3.5/lib/action_controller/params_parser.rb:42:in `parse_formatted_parameters'
/Library/Ruby/Gems/1.8/gems/actionpack-2.3.5/lib/action_controller/params_parser.rb:11:in `call'
/Library/Ruby/Gems/1.8/gems/actionpack-2.3.5/lib/action_controller/session/cookie_store.rb:93:in `call'
/Library/Ruby/Gems/1.8/gems/actionpack-2.3.5/lib/action_controller/failsafe.rb:26:in `call'
/Users/ben/projects/safecell/safecellweb/lib/compressed_requests.rb:36:in `call'
/Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/lock.rb:11:in `call'
/Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/lock.rb:11:in `synchronize'
/Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/lock.rb:11:in `call'
/Library/Ruby/Gems/1.8/gems/actionpack-2.3.5/lib/action_controller/dispatcher.rb:114:in `call'
/Library/Ruby/Gems/1.8/gems/actionpack-2.3.5/lib/action_controller/reloader.rb:34:in `run'
/Library/Ruby/Gems/1.8/gems/actionpack-2.3.5/lib/action_controller/dispatcher.rb:108:in `call'
/Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/rails/rack/static.rb:31:in `call'
/Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/urlmap.rb:46:in `call'
/Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/urlmap.rb:40:in `each'
/Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/urlmap.rb:40:in `call'
/Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/rails/rack/log_tailer.rb:17:in `call'
/Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/content_length.rb:13:in `call'
/Library/Ruby/Gems/1.8/gems/rack-1.0.1/lib/rack/handler/webrick.rb:50:in `service'
One last piece of information, I'm inserting this middleware after ActionController::Failsafe.
EDIT: Looks like it's not a truncation problem
After more digging, it looks like it's not a truncation problem after all. The logs are simply clipping the output so it looks like a truncation issue.
At this point I'm not sure why the JSON is coming in as invalid. Do I need to do any manual escaping?
I'm not a ruby expert by any stretch. I also haven't tried to repro this problem in order to verify my results. But after digging through the rack and actionpack code, I might have something.
The doc for "rack.input" states: "The input stream is an IO-like object which contains the raw HTTP POST data."
So you're using that correctly, it would seem.
However, actionpack tries to parse JSON out of the body (if the content type is specified as JSON) and retrieves the body like this:
when :json
body = request.raw_post
where "request" is actionpack's own Request class, and "raw_post" is defined like this:
def raw_post
unless @env.include? 'RAW_POST_DATA'
@env['RAW_POST_DATA'] = body.read(@env['CONTENT_LENGTH'].to_i)
body.rewind if body.respond_to?(:rewind)
end
@env['RAW_POST_DATA']
end
and "Request.body" is:
def body
if raw_post = @env['RAW_POST_DATA']
raw_post.force_encoding(Encoding::BINARY) if raw_post.respond_to?(:force_encoding)
StringIO.new(raw_post)
else
@env['rack.input']
end
end
That all looks fine and good (though it's confusing to figure out who caches the value first :) ). It looks like the problem is in how the post data is read:
@env['RAW_POST_DATA'] = body.read(@env['CONTENT_LENGTH'].to_i)
So I'm guessing the problem is that since you change "rack.input" but don't update "CONTENT_LENGTH", actionpack is truncating the data since obviously the zipped content would've been shorter than the unzipped content.
Try updating "CONTENT_LENGTH" in your middleware code and see if that fixes it.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With