Node.js virtual memory increases constantly while using child process

Question

I'm using spawn-child npm package to spawn a shell where i run a binary file which was originally built on C++. I provide Stdin's to the binary and then the binary would be sending out the Stdout's constantly for every second. On the node part once i start receiving the Stdout's from binary i have an on listener which would look something like stdout.on('data', function (data) {}) where i send these data's to the SSE channel.

Everything is working fine but the major concern is the constant memory growth of node process that i see when i hit the binary everytime with an Stdin. I have outlined how my code looks, is there an elegant way to control this memory growth, if so please share.

var sseChannel = require('sse-channel'),
    spawnCommand = require('spawn-command'),
    cmd = 'path to the binary file',
    globalArray = [],
    uuid = require('uuid');

module.exports = function(app) {
  var child = spawnCommand(cmd),
    privateChannel = new sseChannel({
      historySize: 0,
      cors: {
        origins: ['*']
      },
      pingInterval: 15 * 1000,
      jsonEncode: false
    });
  srvc = {
    get: function(req, res) {
      globalArray[uuid.v4()] = res;
      child.stdin.write('a json object in a format that is expected by binary' + '\n'); // req.query.<queryVal>
      child.stdout.on('data', function(data) {
        privateChannel.send(JSON.stringify(data));
      });
    },
    delete: function(sessionID) {
       var response = globalArray[sessionID];
       privateChannel.removeClient(response);
       response.end();
       delete globalArray[sessionID];
    }
  }
}

This code is just to enumerate how it would look in the app. Hitting the Run code snippet would not work in this case.

I collected heapdump at 2 different intervals and this is how the statistics looks, there is a tremendous increase in the Typed Array value, what could be done to maintain or suppress the growth of Typed Array,

Answer 1

The problem is that you're spawning a process once and then adding a new data event handler for every request to your http server that never gets removed. So this would explain why the memory usage never drops even after gc.

Another (unrelated) problem is that if you are using your single child process to process multiple incoming requests, you can run into the problem of mixing responses for different requests (you cannot assume that one data event will contain only the data for a particular request). If the child process is node.js-based, you could set up an ipc channel with it and then just pass regular JavaScript values back and forth instead of setting up stdout handling/parsing. If the child isn't node.js-based or you want an alternative (no-ipc) solution, you could set up a queue that all requests get pushed onto and then have a function that processes the queue and responds to each request serially (only moving onto the next request once you have somehow determined you have received all output from the child process for the current request).

If you instead meant for the child process to only be used for a single request, you will need to tweak your code to spawn once per request instead (moving spawn() inside get()).