Run tasks in multiple threads on node.js

Question

I need some help with node.js as i'm a newbie to it. I have an array of data and have to run functions with that data in multiple threads (using all CPU cores). In usual for me languages i do create threadpool with number of threads, push some tasks to it, wait until finish, send more tasks to queue. But i can't figure it out how to do that in nodejs.

PS: i'm using latest 0.8 branch.

Answer 1

Nodejs is built to run on a single process, but you can spawn other processes. The cluster module uses the fork method from the child_process module, but it's intended for distributing connections of a server across processes and sharing the same port.

I think you want exec. Example:

var exec = require('child_process').exec,
child;

var array = ["a", "b", "c", "d", "e"]; // your array of data

var n = array.length; // length
var done = 0; // jobs done
var i = n; // iterator

while(i--) { // reverse while loops are faster
    (function (argument) { // A closure
        child = exec('node otherFunction.js '+argument, // Spawn the process, with an item from your array as an argument.
          function (error, stdout, stderr) {
            if (error === null) {
                done += 1;
                if (done === n) {
                    console.log('Everything is done');
                }
            }
        });
    })(array[i]);
}

The above is of course bad code and not even tested, but I think it would work. All you have to do is call the function you wanted to call on the array's items in otherFunction.js, inside that you would find the arguments in process.argv.