Having trouble streaming response to client using expressjs

Question

I am having a really hard time wrapping my head around how to stream data back to my client when using Nodejs/Expressjs.

I am grabbing a lot of data from my database and I am doing it in chunks, I would like to stream that back to the client as I get the data such that I do not have to store the entire dataset in memory as a json object before sending it back.

I would like the data to stream back as a file, ie I want the browser to ask my users what to do with the file on download. I was previously creating a file system write stream and stream the contents of my data to the file system, then when done I would send the file back to the client. I would like to eliminate the middle man (creating tmp file on file system) and just stream data to client.

app.get(
    '/api/export',
    function (req, res, next) {
        var notDone = true;
        while (notDone) {
            var partialData = // grab partial data from database (maybe first 1000 records);

            // stream this partial data as a string to res???

            if (checkIfDone) notDone = false;
        }
    }
);

I can call res.write("some string data"), then call res.end() when I am done. However I am not 100% sure that this is actually streaming the response to the client as I write. Seems like expressjs is storing all the data until I call end and then sending the response. Is that true?

What is the proper way to stream strings chunks of data to a response using expressjs?

Answer 1

The response object is already a writable stream. Express handles sending chunked data automatically, so you won't need to do anything extra but:

response.send(data)

You may also want to check out the built-in pipe method, http://nodejs.org/api/stream.html#stream_event_pipe.