Node.js: can you use asynchronous functions from within streams?

Question

Consider the following:

var asyncFunction = function(data, callback) {   doAsyncyThing(function(data){     // do some stuff     return callback(err)   }) } fs.createReadStream('eupmc_lite_metadata_2016_04_15.json')   .pipe(JSONstream.parse())   .on('data', asyncFunction)   // <- how to let asyncFunction complete before continuing 

How does the stream know when asyncFunction has completed? Is there any way to use asynchronous functions from within streams?

Answer 1

Check out transform streams. They give you the ability to run async code on a chunk, and then call a callback when you are finished. Here are the docs: https://nodejs.org/api/stream.html#stream_transform_transform_chunk_encoding_callback

As a simple example, you can do something like:

const Transform = require('stream').Transform class WorkerThing extends Transform {     _transform(chunk, encoding, cb) {         asyncFunction(chunk, cb)     } }  const workerThing = new WorkerThing()  fs.createReadStream('eupmc_lite_metadata_2016_04_15.json') .pipe(JSONstream.parse()) .pipe(workerThing)