nodejs - parsing chunked twitter json

Question

The nodejs server 'gets' this JSON stream from Twitter and sends it to the client:

stream.twitter.com/1/statuses/filter.json?track=gadget

The data returned to the client is 'chunked' JSON and both JSON.parse(chunk) and eval('(' + chunk + ')') on the client side result in parsing errors. Concatenating the chucked pieces and waiting for the 'end' event isn't a solution either

I noticed previous samples used something like this on the client side that apparently worked before:

  socket.onmessage = function(chunk) { 
  data = eval("(" + chunk.data + ")");
  alert(data.user.screen_name);

I'm using this on the client side and it results in a parsing error:

var socket = new io.Socket();
    socket.on('message', function(chunk) { 
    var data = eval('(' + chunk + ')'); // parsing error
    alert(data.screen_name): 

I know that its successfully returning a JSON chunk with:

  var socket = new io.Socket();
        socket.on('message', function(chunk) {  
        alert(chunk): // shows a JSON chunk

Server:

  response.on('data', function (chunk) {
    client.each(function(e) {
      e.send(chunk);  
  });  

Did something change or what else em I doing wrong?

UPDATE: The 'end' event does not fire because its streaming?

http.get({
  headers: { 'content-type': 'application/json' },
  host: 'stream.twitter.com',
  path: '/1/statuses/filter.json?track...
}, function(res) {

  res.setEncoding('utf8');
  res.on('data', function (chunk) {
    client.each(function(e) {
      e.send(chunk);  
  });  
 });

  // does not fire
  res.on('end', function () {

  });  

...

I'm looking into the difference with http 1.0 and http 1.1 as far as sending chunked data.

Answer 1

Look at the section titled Parsing Responses in Twitter's documentation.

Parsing JSON responses from the Streaming API is simple every object is returned on its own line, and ends with a carriage return. Newline characters (\n) may occur in object elements (the text element of a status object, for example), but carriage returns (\r) should not.

On the server side, keep accumulating chunks until you see the carriage return "\r". Once the carriage return is found, extract the string up to the carriage return, and that gives us one tweet.

var message = ""; // variable that collects chunks
var tweetSeparator = "\r";

res.on('data', function(chunk) {
    message += chunk;

    var tweetSeparatorIndex = message.indexOf(tweetSeparator);
    var didFindTweet = tweetSeparatorIndex != -1;

    if (didFindTweet) {
        var tweet = message.slice(0, tweetSeparatorIndex);
        clients.forEach(function(client) {
            client.send(tweet);
        });
        message = message.slice(tweetSeparatorIndex + 1);
    }
});

The client becomes simple. Simply parse the socket message as JSON in its entirety.

socket.on('message', function(data) {
    var tweet = JSON.parse(data);
});

Answer 2

@Anurag I'cant add comments, however instead of

if (chunk.substr("-1") == "\r") 

it should be:

if ( chunk.charCodeAt(chunk.length-2) == 13 )

The carriage return isn't the last character.