Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Streaming API vs Rest API?

The canonical example here is Twitter's API. I understand conceptually how the REST API works, essentially its just a query to their server for your particular request in which you then receive a response (JSON, XML, etc), great.

However I'm not exactly sure how a streaming API works behind the scenes. I understand how to consume it. For example with Twitter listen for a response. From the response listen for data and in which the tweets come in chunks. Build up the chunks in a string buffer and wait for a line feed which signifies end of Tweet. But what are they doing to make this work?

Let's say I had a bunch of data and I wanted to setup a streaming API locally for other people on the net to consume (just like Twitter). How is this done, what technologies? Is this something Node JS could handle? I'm just trying to wrap my head around what they are doing to make this thing work.

like image 911
aherrick Avatar asked May 11 '11 06:05

aherrick


People also ask

What is streaming API?

Streaming API is a subscription mechanism based on CometD, which enables real-time streaming of event messages. CometD enables the server to push data to the client when the data is available and while the client maintains a connection to the server.

Does rest support streaming?

It supports bidirectional streaming.

What is opposite of REST API?

Streaming APIs are almost an exact opposite of the REST ethos. In its most basic state, Streaming APIs invert the conversational nature of REST, where a request is made and a response is given, and instead has the server send information to a client when an update is ready.


1 Answers

Twitter's stream API is that it's essentially a long-running request that's left open, data is pushed into it as and when it becomes available.

The repercussion of that is that the server will have to be able to deal with lots of concurrent open HTTP connections (one per client). A lot of existing servers don't manage that well, for example Java servlet engines assign one Thread per request which can (a) get quite expensive and (b) quickly hits the normal max-threads setting and prevents subsequent connections.

As you guessed the Node.js model fits the idea of a streaming connection much better than say a servlet model does. Both requests and responses are exposed as streams in Node.js, but don't occupy an entire thread or process, which means that you could continue pushing data into the stream for as long as it remained open without tying up excessive resources (although this is subjective). In theory you could have a lot of concurrent open responses connected to a single process and only write to each one when necessary.

If you haven't looked at it already the HTTP docs for Node.js might be useful.

I'd also take a look at technoweenie's Twitter client to see what the consumer end of that API looks like with Node.js, the stream() function in particular.

like image 126
Richard Marr Avatar answered Sep 22 '22 00:09

Richard Marr