What is the `pipe` method in NodeJS?

Question

I see this method in use in many cases but I cannot understand what it does.

An example:

var net = require('net');
var server = net.createServer(function(connection) { 
   console.log('client connected');

   connection.on('end', function() {
      console.log('client disconnected');
   });


   connection.write('Hello World!\r\n');
   connection.pipe(connection);
});

server.listen(8080, function() { 
   console.log('server is listening');
});

When is the connection.pipe(connection); do?

What I commented this line out, the behavior sees the same.

Another example:

var fs = require("fs");
var zlib = require('zlib');

// Compress the file input.txt to input.txt.gz
fs.createReadStream('input.txt')
   .pipe(zlib.createGzip())
   .pipe(fs.createWriteStream('input.txt.gz'));

console.log("File Compressed.");

Can anybody explain what does the pipe method do?

The documentation is not clear. thanks in advance.

Answer 1

.pipe() feeds a readable stream into a writable stream so that as the contents are incrementally read from the readable stream, they are automatically forwarded to the writable stream. It manages the reading from one stream and writing to another stream automatically for you.

In your first example:

connection.pipe(connection)

it reads from the incoming stream of the TCP connection and writes that back to the outgoing stream which would essentially echo whatever was sent to the server back to the sender. So, if you sent "abc" to this server, you would get back as the response the same "abc" that you sent. This can only work if the connection object is both a readable and a writable stream (which it is in this case).

Your second example:

// Compress the file input.txt to input.txt.gz
fs.createReadStream('input.txt')
   .pipe(zlib.createGzip())
   .pipe(fs.createWriteStream('input.txt.gz'));

Reads from input.txt, creates a Gzip stream and passes the data from input.txt into the Gzip stream which is then passed to a writable stream in the file input.txt.gz. The next result would be to take the contents of input.txt, GZip compress them and write them to input.txt.gz, but it's all done incrementally with streams so you could process very large files without ever reading more than a little bit at a time into memory.

The streams involved in these operations handle all the buffering, error handling and automatic closing of files.

Answer 2

pipe is common used in stream. Why we need stream? The simple example I can think of is for handling big file. Let's say we have a file 10,000 rows or more. Instead of loading all the rows into memory (which may cause memory limit error), we can use stream here to process and load the row one by one.

fs.createReadStream('input.txt')
   .pipe(zlib.createGzip())
   .pipe(fs.createWriteStream('input.txt.gz'));

This is example where you create a gzip file using stream. The content of input.txt is piped to gzip function then piped to output file input.txt.gz.

Good reference about stream https://github.com/substack/stream-handbook

Hope it helps.