Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What's the difference between .pipe and pipeline on streams

Tags:

stream

node.js

I found two different ways to pipe streams in node.js

Well known .pipe() method of a stream

https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options

and standalone function for streams

https://nodejs.org/api/stream.html#stream_stream_pipeline_streams_callback

Which one should I use and what are the benefits between those two?

like image 687
QuestionAndAnswer Avatar asked Nov 15 '19 11:11

QuestionAndAnswer


People also ask

What is a stream pipeline?

Stream operations are divided into intermediate and terminal operations, and are combined to form stream pipelines. A stream pipeline consists of a source (such as a Collection , an array, a generator function, or an I/O channel); followed by zero or more intermediate operations such as Stream. filter or Stream.

What is the difference between piping and pipeline piping vs pipeline?

Piping is normally connected with various equipments and carry fluids inside a complex network that will be processed in that equipment. Whereas Pipelines supply the feed for further processing or deliver the processed fluid or end product and they normally run straight.

How is a pipe different from and related to a stream?

In most cases, clists and streams are used to implement a portion of the link between a user process (specifically, a file descriptor) and a character device driver (for example, a serial port or a pty). Pipes are also a queue, but they link user processes (specifically, a pair of file descriptors).

What is the correct way to pipe a readable stream?

To consume a readable stream, we can use the pipe / unpipe methods, or the read / unshift / resume methods. To consume a writable stream, we can make it the destination of pipe / unpipe , or just write to it with the write method and call the end method when we're done.


3 Answers

TL;DR - You better want to use pipeline

What's pipeline?

From the docs: A module method to pipe between streams forwarding errors and properly cleaning up and provide a callback when the pipeline is complete.

What's the motivation for using pipeline?

❌ Let's take a look at the following code:

const { createReadStream } = require('fs');
const { createServer } = require('http');
const server = createServer(
  (req, res) => {
    createReadStream(__filename).pipe(res);
  }
);

server.listen(3000);

What's wrong here? If the response will quit or the client closes the connection - then the read stream is not closed or destroy which leads to a memory leak.

✅So if you use pipeline, it would close all other streams and make sure that there are no memory leaks.

const { createReadStream } = require('fs');
const { createServer } = require('http');
const { pipeline } = require('stream');

const server = createServer(
  (req, res) => {
    pipeline(
      createReadStream(__filename),
      res,
      err => {
        if (err)
          console.error('Pipeline failed.', err);
        else
          console.log('Pipeline succeeded.');
      }
    );
  }
);

server.listen(3000);
like image 90
Idan Dagan Avatar answered Oct 11 '22 07:10

Idan Dagan


pipeline is the improved version of pipe it was added to stream module since Node.js v10

Also, pipeline takes any number of arguments and the last argument is a callback used to know when the pipeline ends or throws an error.

Usage pipe:

mySourceStream.pipe(myStream).pipe(anotherStream)

Usage pipeline:

mySourceStream.pipeline(myStream, anotherStream, err => {
  if (err) {
    console.log('There is an error')
  } else {
    console.log('pipeline successful')
  }
})
like image 23
Leonardo Rivera Avatar answered Oct 11 '22 08:10

Leonardo Rivera


According to the documentation, they both do the same thing. But there some differences:

  • .pipe() is a method of Readable, while pipeline is a module method of stream that accepts streams to pipe.
  • pipeline() method provide a callback when the pipeline is complete.
  • pipeline() method was added since 10 version of node, while .pipe exist from the earliest versions of Node.

In my opinion, with pipeline() code looks a bit cleaner, but you can use both of them.

Example of .pipe():

const fs = require('fs');
const r = fs.createReadStream('file.txt');
const z = zlib.createGzip();
const w = fs.createWriteStream('file.txt.gz');
r.pipe(z).pipe(w);

Example of pipeline():

const { pipeline } = require('stream');
const fs = require('fs');
const zlib = require('zlib');

pipeline(
  fs.createReadStream('archive.tar'),
  zlib.createGzip(),
  fs.createWriteStream('archive.tar.gz'),
  (err) => {
    if (err) {
      console.error('Pipeline failed.', err);
    } else {
      console.log('Pipeline succeeded.');
    }
  }
);
like image 6
Poul Uvarov Avatar answered Oct 11 '22 07:10

Poul Uvarov