HTTP streaming in rails not working when using Rack::Deflater

Question

I've setup unicorn in rails 3.1 and http streaming works until I enable Rack::Deflater. I've tried both with and without use Rack::Chunked. In curl I can see my response while in chrome I get the following errror: ERR_INVALID_CHUNKED_ENCODING

The result is same in other browsers (firefox, safari) and between development (osx) and production (heroku).

config.ru:

require ::File.expand_path('../config/environment',  __FILE__)
use Rack::Chunked
use Rack::Deflater
run Site::Application

unicorn.rb:

listen 3001, :tcp_nopush => false
worker_processes 1 # amount of unicorn workers to spin up
timeout 30         # restarts workers that hang for 30 seconds

controller:

render "someview", :stream => true

Thanks for any help.

Answer 1

The problem is that Rails ActionController::Streaming renders directly into a Chunked::Body. This means the content is first chunked and then gzipped by the Rack::Deflater middleware, instead of gzipped and then chunked.

According to the HTTP/1.1 RFC 6.2.1, chunked must be last applied encoding to a transfer.

Since "chunked" is the only transfer-coding required to be understood by HTTP/1.1 recipients, it plays a crucial role in delimiting messages on a persistent connection. Whenever a transfer-coding is applied to a payload body in a request, the final transfer-coding applied must be "chunked".

I fixed it for us by monkey patching ActionController::Streaming _process_options and _render_template methods in an initializer so it does not wrap the body in a Chunked::Body, and lets the Rack::Chunked middleware do it instead.

module GzipStreaming
  def _process_options(options)
    stream = options[:stream]
    # delete the option to stop original implementation  
    options.delete(:stream)
    super
    if stream && env["HTTP_VERSION"] != "HTTP/1.0"
      # Same as org implmenation except don't set the transfer-encoding header
      # The Rack::Chunked middleware will handle it 
      headers["Cache-Control"] ||= "no-cache"
      headers.delete('Content-Length')
      options[:stream] = stream
    end
  end

  def _render_template(options)
    if options.delete(:stream)
      # Just render, don't wrap in a Chunked::Body, let
      # Rack::Chunked middleware handle it
      view_renderer.render_body(view_context, options)
    else
      super
    end
  end
end

module ActionController
  class Base
    include GzipStreaming
  end
end

And leave your config.ru as

require ::File.expand_path('../config/environment',  __FILE__)
use Rack::Chunked
use Rack::Deflater
run Roam7::Application

Not a very nice solution, it will probably break some other middlewares that inspect/modify the body. If someone has a better solution I'd love to hear it.

If you are using new relic, its middleware must also be disabled when streaming.