How to pipe JSON to function in node.js?

Question

I asked this question:

How to pipe to function in node.js?

But I find that it's not as easy when using JSON. Because the JSON is streamed in chunks, it breaks when you try to JSON.parse a chunk that does not represent a complete JSON object.

The following code works fine for small JSON files but breaks for large JSON files. How do I do this when streaming large JSON files?

const fs = require('fs')
const {Transform} = require('stream')

const jsonTransform = new Transform({
    readableObjectMode: true,
    writableObjectMode: true,
    transform: (chunk, encoding, done) => {
        let json = JSON.parse(chunk)
        // manipulate json here
        done(null, JSON.stringify(json, null, 2))
    }
})

fs.createReadStream('input.json')
    .pipe(jsonTransform)
    .pipe(fs.createWriteStream('output.json'))

Answer 1

There are a tonne of modules on npm handling this, e.g. JSONstream by Dominic Tarr https://github.com/dominictarr/JSONStream

Another approach is to ensure the json in your input.json is newline delimited (ndjson) and then you can use https://github.com/maxogden/ndjson to parse each line into an object and you can pipe that further to something modifying the objects and stringifying the contents.