node.js stream array of json to response

Question

I have a REST method that should return a JSON array with certain elements that are read from mongodb (using mongoose).

It should be very simple (in the real case the find method has arguments, but that's no the problem):

OutDataModel.find().stream({transform: JSON.stringify}).pipe(res);

The problem with this approach is that I don't get a valid JSON as the result is like this :

{"id":"1","score":11}{"id":"2","score":12}{"id":"3","score":13}

and I was expecting this :

[{"id":"1","score":11},{"id":"2","score":12},{"id":"3","score":13}]

I have not been able to find a solution, but I am pretty sure there will be a simple one.

What I've tried :

Nothing that I'm proud of but here it goes.

1. before streaming write '[' to the response.
2. Instead of JSON.stringify I provided another method that calls JSON.stringify and adds a ',' at the end
3. in the 'end' event of the stream, I write ']' to the response.

Still is not working with this "solution" as I have a trailing comma at the end like this :

 [{"id":"1","score":11},{"id":"2","score":12},{"id":"3","score":13},]

As I said I am pretty sure there should be a clean solution as it is something that should be quite common.

There will be a lot of concurrent invocations to this method, so I don't want to read everything into memory and then write everything to the response. Each of the invocations will not return a lot of records, but all of them together can be huge. The consumer is a java application with spring, using jackson to parse the JSON.

Please let me know how to do it.

ANSWER

I got it working, by creating a Transform stream as was suggested in the accepted answer.

My stream looks like this :

var arraystream = new stream.Transform({objectMode: true});
arraystream._hasWritten = false;


arraystream._transform = function (chunk, encoding, callback) {
    console.log('_transform:' + chunk);
    if (!this._hasWritten) {
        this._hasWritten = true;
        this.push('[' + JSON.stringify(chunk));

    } else {
        this.push(',' + JSON.stringify(chunk));
    }
    callback();
};

arraystream._flush = function (callback) {
    console.log('_flush:');
    this.push(']');
    callback();

};

and the code to use it :

OutDataModel.find().stream().pipe(arraystream).pipe(res);

Thanks.

Answer 1

You're on the right track by implementing your own logic. You could also make use of the ArrayFormatter here which is doing something similar: https://gist.github.com/aheckmann/1403797

The transform function gets called on each document individually -- a Mongoose QueryStream emits a single document per 'data' event but QueryStream isn't treating them semantically as part of any larger array data structure; to get an array-formatted JSON, you really do have to do it yourself (as you have surmised).

Answer 2

I found a very simple and clean solution here: convert mongoose stream to array

That's my simplified version of the code:

Products
    .find({})
    .lean()
    .stream({
        transform: () => {
            let index = 0;
            return (data) => {
                return (!(index++) ? '[' : ',') + JSON.stringify(data);
            };
        }() // invoke
    })
    .on('end', () => {
        res.write(']');
    })
    .pipe(res);