How can I 'accumulate' a raw stream in Node.js?

Question

At the moment I concatenate everything into a string as follows

var body = '';
res.on('data', function(chunk){
    body += chunk;
});

How can I preserve and accumulate the raw stream so I can pass raw bytes to functions that are expecting bytes and not String?

Answer 1

Better use Buffer.concat - much simpler. Available in Node v0.8+.

var chunks = [];
res.on('data', function(chunk) { chunks.push(chunk); });
res.on('end', function() {
    var body = Buffer.concat(chunks);
    // Do work with body.
});

Answer 2

First off, check that these functions actually need the bytes all in one go. They really should accept 'data' events so that you can just pass on the buffers in the order you receive them.

Anyway, here's a bruteforce way to concatenate all data chunk buffers without decoding them:

var bodyparts = [];
var bodylength = 0;
res.on('data', function(chunk){
    bodyparts.push(chunk);
    bodylength += chunk.length;
});
res.on('end', function(){
    var body = new Buffer(bodylength);
    var bodyPos=0;
    for (var i=0; i < bodyparts.length; i++) {
        bodyparts[i].copy(body, bodyPos, 0, bodyparts[i].length);
        bodyPos += bodyparts[i].length;
    }
    doStuffWith(body); // yay
});

Answer 3

Alternately, you can also use a node.js library like bl or concat-stream:

'use strict'
let http = require('http')
let bl = require('bl')
http.get(url, function (response) {
  response.pipe(bl(function (err, data) {
    if (err)
      return console.error(err)
    console.log(data)
  }))
})