How do pipe (stream of Node.js) and bl (BufferList) work together?

Question

This is actually the exercise No.8 from the Node.js tutorial ([https://github.com/workshopper/learnyounode][1])

The goal: Write a program that performs an HTTP GET request to a URL provided to you as the first command-line argument. Collect all data from the server (not just the first "data" event) and then write two lines to the console (stdout).

The first line you write should just be an integer representing the number of characters received from the server. The second line should contain the complete String of characters sent by the server.

So here's my solution(It passes but looks uglier compared to the official solution).

var http = require('http'),
bl = require('bl');

var myBL = new bl(function(err, myBL){
    console.log(myBL.length);
    console.log(myBL.toString());
});

var url = process.argv[2];
http.get(url, function(res){
    res.pipe(myBL);
    res.on('end', function(){
        myBL.end();
    });
});

The official solution:

var http = require('http')
var bl = require('bl')

http.get(process.argv[2], function (response) {
    response.pipe(bl(function (err, data) {
        if (err)
            return console.error(err)
        data = data.toString()
        console.log(data.length)
        console.log(data)
    }))
})

I have difficulties understanding how the official solution works. I have mainly two questions:

1. The bl constructor expects the 2nd argument to be an instance of bl (according to bl module's documentation, [https://github.com/rvagg/bl#new-bufferlist-callback--buffer--buffer-array-][2]), but what is data? It came out of nowhere. It should be undefined when it is passed to construct the bl instance.

2. when is bl.end() called? I can see no where that the bl.end() is called...

Hope someone can shed some light on these questions. (I know I should've read the source code, but you know...)

  [1]: https://github.com/workshopper/learnyounode
  [2]: https://github.com/rvagg/bl#new-bufferlist-callback--buffer--buffer-array-

Answer 1

This portion of the bl github page more or less answers your question:

Give it a callback in the constructor and use it just like concat-stream:

const bl = require('bl')
, fs = require('fs')

fs.createReadStream('README.md')   
   .pipe(bl(function (err, data) { //  note 'new' isn't strictly required
      // `data` is a complete Buffer object containing the full data
      console.log(data.toString())   
}))

Note that when you use the callback method like this, the resulting data parameter is a concatenation of all Buffer objects in the list. If you want to avoid the overhead of this concatenation (in cases of extreme performance consciousness), then avoid the callback method and just listen to 'end' instead, like a standard Stream.

You're passing a callback to bl, which is basically a function that it will call when it has a stream of data to do something with. Thus, data is undefined for now... it's just a parameter name that will later be used to pass the text from the GET call for printing.

I believe that bl.end() doesn't have be called because there's no real performance overhead to letting it run, but I could be wrong.

Answer 2

I have read the source code of bl library and node stream API.

BufferList is a custom duplex stream,that is both Readable and Writable.When you run readableStream.pipe(BufferList), by default end() is called on BufferList as the destination when the source stream emits end() which fires when there will be no more data to read.

See the implementation of BufferList.prorotype.end:

BufferList.prototype.end = function (chunk) {
  DuplexStream.prototype.end.call(this, chunk)

  if (this._callback) {
    this._callback(null, this.slice())
    this._callback = null
  }
}

So the callback passed to BufferList, will be called after BufferList received all data from the source stream, call this.slice() will return the result of concatenating all the Buffers in the BufferList where is the data parameter comes from.