Running out of memory writing to a file in NodeJS

Question

I'm processing a very large amount of data that I'm manipulating and storing it in a file. I iterate over the dataset, then I want to store it all in a JSON file.

My initial method using fs, storing it all in an object then dumping it didn't work as I was running out of memory and it became extremely slow.

I'm now using fs.createWriteStream but as far as I can tell it's still storing it all in memory.

I want the data to be written object by object to the file, unless someone can recommend a better way of doing it.

Part of my code:

  // Top of the file
  var wstream = fs.createWriteStream('mydata.json');
  ...

  // In a loop
  let JSONtoWrite = {}
  JSONtoWrite[entry.word] = wordData

  wstream.write(JSON.stringify(JSONtoWrite))

  ...
  // Outside my loop (when memory is probably maxed out)
  wstream.end()

I think I'm using Streams wrong, can someone tell me how to write all this data to a file without running out of memory? Every example I find online relates to reading a stream in but because of the calculations I'm doing on the data, I can't use a readable stream. I need to add to this file sequentially.

Answer 1

The problem is that you're not waiting for the data to be flushed to the filesystem, but instead keep throwing new and new data to the stream synchronously in a tight loop.

Here's an piece of pseudocode that should work for you:

    // Top of the file
    const wstream = fs.createWriteStream('mydata.json');
    // I'm no sure how're you getting the data, let's say you have it all in an object
    const entry = {};
    const words = Object.keys(entry);

    function writeCB(index) {
       if (index >= words.length) {
           wstream.end()
           return;
       }

       const JSONtoWrite = {};
       JSONtoWrite[words[index]] = entry[words[index]];   
       wstream.write(JSON.stringify(JSONtoWrite), writeCB.bind(index + 1));
    }

    wstream.write(JSON.stringify(JSONtoWrite), writeCB.bind(0));