Spawning child processes in an express server

Question

In my express server there are some functions which I need to run as child processes because otherwise they'll tie up the server and other people won't be able to access it. They're already using the async module but they still tie up the server unless they're run as child processes.

One problem is passing the req and res parameters to them.

How can this be done?

Answer 1

Using child_process.fork, you can send messages to child processes.

Edit: I incorrectly advised to pass req and res as message parameters to the child process. This is not possible, as all messages to and from child processes are converted to JSON. Instead, you could keep some kind of queue in your server. The below is only meant as an example, you may want something more robust:

child.js:

process.on('message', function(message) {
    // Process data

    process.send({id: message.id, data: 'some result'});
});

server.js:

var child_process = require('child_process');
var child = child_process.fork(__dirname + '/child.js');
var taskId = 0;
var tasks = {};

function addTask(data, callback) {
    var id = taskId++;

    child.send({id: id, data: data});

    tasks[id] = callback;
};

child.on('message', function(message) {
    // Look up the callback bound to this id and invoke it with the result
    tasks[message.id](message.data);
});

app.post('/foo', function(req, res) {
    addTask('some data', function(result) {
        res.send(result);
    });
});

It's a bit more involved, but it should work. You may quickly grow out of such a system, and may be better served by a proper queue.