A client for HTTP server push (streaming) in Rust?

Question

For the lack of a better example, let's say I want to write a simple client with Rust that could establish a connection and receive data from Twitter's HTTP Streaming API. Is this possible yet? I've been keeping an eye on Iron and Nickel which seem like good frameworks, but I don't think they have this feature yet?

Answer 1

The http client hyper supports reading responses incrementally (as anything that implements rust's Reader trait), but I wasn't able to find anything to parse the response incrementally, or that implements twitter's particular protocol (to end objecs with \r\n).

That said, I was able to implement a quick'n'dirty proof of concept:

EDIT: See and play with it on github.

use rustc_serialize::json::Json;
use std::str;

pub trait JsonObjectStreamer {
    fn json_objects(&mut self) -> JsonObjects<Self>;
}

impl<T: Buffer> JsonObjectStreamer for T {
    fn json_objects(&mut self) -> JsonObjects<T> {
        JsonObjects { reader: self }
    }
}

pub struct JsonObjects<'a, B> where B: 'a {
    reader: &'a mut B
}

impl<'a, B> Iterator for JsonObjects<'a, B> where B: Buffer + 'a {
    type Item = Json;

    fn next(&mut self) -> Option<Json> {
        let mut line_bytes = match self.reader.read_until(b'\r') {
            Ok(bytes) => bytes,
            Err(_)    => return None,
        };

        if line_bytes.last() == Some(&b'\r') {
            // drop the \r
            line_bytes.pop();

            // skip the \n
            match self.reader.read_char() {
                Ok(_)  => (),
                Err(_) => return None,
            }
        }

        let line = match str::from_utf8(&line_bytes) {
            Ok(line) => line,
            Err(_)   => return None
        };

        Json::from_str(line).ok()
    }
}

Usage: (assuming you have dropped it on a src/json_streamer.rs file on your project)

#![feature(io)]

extern crate hyper;
extern crate "rustc-serialize" as rustc_serialize;

mod json_streamer;

use hyper::Client;

use std::old_io::BufferedReader;
use json_streamer::JsonObjectStreamer;

fn main() {
    let mut client = Client::new();
    let res = client.get("http://localhost:4567/").send().unwrap();

    for obj in BufferedReader::new(res).json_objects() {
        println!("object arrived: {}", obj);
    }
}

I've used this tiny sinatra app to test it:

require 'sinatra'
require 'json'

class Stream
  def each
    hash = { index: 0 }

    loop do
      hash[:index] += 1
      yield hash.to_json + "\r\n"
      sleep 0.5
    end
  end
end

get '/' do
  Stream.new
end