I found two different ways to pipe streams in node.js
Well known .pipe()
method of a stream
https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options
and standalone function for streams
https://nodejs.org/api/stream.html#stream_stream_pipeline_streams_callback
Which one should I use and what are the benefits between those two?
Stream operations are divided into intermediate and terminal operations, and are combined to form stream pipelines. A stream pipeline consists of a source (such as a Collection , an array, a generator function, or an I/O channel); followed by zero or more intermediate operations such as Stream. filter or Stream.
Piping is normally connected with various equipments and carry fluids inside a complex network that will be processed in that equipment. Whereas Pipelines supply the feed for further processing or deliver the processed fluid or end product and they normally run straight.
In most cases, clists and streams are used to implement a portion of the link between a user process (specifically, a file descriptor) and a character device driver (for example, a serial port or a pty). Pipes are also a queue, but they link user processes (specifically, a pair of file descriptors).
To consume a readable stream, we can use the pipe / unpipe methods, or the read / unshift / resume methods. To consume a writable stream, we can make it the destination of pipe / unpipe , or just write to it with the write method and call the end method when we're done.
TL;DR - You better want to use pipeline
What's pipeline?
From the docs: A module method to pipe between streams forwarding errors and properly cleaning up and provide a callback when the pipeline is complete.
What's the motivation for using pipeline?
❌ Let's take a look at the following code:
const { createReadStream } = require('fs');
const { createServer } = require('http');
const server = createServer(
(req, res) => {
createReadStream(__filename).pipe(res);
}
);
server.listen(3000);
What's wrong here? If the response will quit or the client closes the connection - then the read stream is not closed or destroy which leads to a memory leak.
✅So if you use pipeline
, it would close all other streams and make sure that there are no memory leaks.
const { createReadStream } = require('fs');
const { createServer } = require('http');
const { pipeline } = require('stream');
const server = createServer(
(req, res) => {
pipeline(
createReadStream(__filename),
res,
err => {
if (err)
console.error('Pipeline failed.', err);
else
console.log('Pipeline succeeded.');
}
);
}
);
server.listen(3000);
pipeline is the improved version of pipe it was added to stream module since Node.js v10
Also, pipeline takes any number of arguments and the last argument is a callback used to know when the pipeline ends or throws an error.
Usage pipe:
mySourceStream.pipe(myStream).pipe(anotherStream)
Usage pipeline:
mySourceStream.pipeline(myStream, anotherStream, err => {
if (err) {
console.log('There is an error')
} else {
console.log('pipeline successful')
}
})
According to the documentation, they both do the same thing. But there some differences:
.pipe()
is a method of Readable
, while pipeline
is a module method of stream
that accepts streams to pipe.pipeline()
method provide a callback when the pipeline is complete.pipeline()
method was added since 10 version of node, while .pipe
exist from the earliest versions of Node. In my opinion, with pipeline()
code looks a bit cleaner, but you can use both of them.
Example of .pipe()
:
const fs = require('fs');
const r = fs.createReadStream('file.txt');
const z = zlib.createGzip();
const w = fs.createWriteStream('file.txt.gz');
r.pipe(z).pipe(w);
Example of pipeline()
:
const { pipeline } = require('stream');
const fs = require('fs');
const zlib = require('zlib');
pipeline(
fs.createReadStream('archive.tar'),
zlib.createGzip(),
fs.createWriteStream('archive.tar.gz'),
(err) => {
if (err) {
console.error('Pipeline failed.', err);
} else {
console.log('Pipeline succeeded.');
}
}
);
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With