Streaming data from Sinatra/Rack application

Question

I am trying to stream textual data (XML/JSON) from a Ruby (1.9.1p378) Sinatra (1.0) Rack (1.2.1) application. The suggested solutions (e.g. Is there a way to flush html to the wire in Sinatra) do not seem to work - the server just blocks when I yield elements of some infinite stream (e.g. from %w(foo bar).cycle). I tried webrick and thin as servers.

Any suggestions on getting this done? Should I use http://sinatra.rubyforge.org/api/classes/Sinatra/Streaming.html and if so how would I use it in my application?

Answer 1

Starting with Sinatra 1.3, you could also use the new streaming API:

get '/evented' do   stream(:keep_open) do |out|     EventMachine::PeriodicTimer.new(1) { out << "#{Time.now}\n" }   end end 

Answer 2

As Colin mentioned, Goliath can stream response data, as well as incoming (large file uploads). There is an example in the repo for streaming data to the client: https://github.com/postrank-labs/goliath/blob/master/examples/stream.rb

Instead of a timer, you can easily hook up any other data stream to push data to the client. For example, you can connect an AMQP queue, or any other message queue directly to Goliath and let it act as an HTTP frontend to that data.

Answer 3

You should definitely take a look at the rack-able Goliath web server. It supports streaming out-of-the-box. I am using it for a firehose style streaming api.

Goliath is both an app server and a lightweight framework designed to meet the following goals: fully asynchronous processing, middleware support, simple configuration, high-performance, and arguably most importantly, readable and maintainable code.