How do i execute multiple nodejs processes in parallel

Question

i'm using nodejs to capture a video stream and then calling ffmpeg using spawn to process the feed. I would like to do this in parallel if i receive multiple video streams.

I can mimic this manually by opening multiple terminals and using a different variation of the ffmpeg command to execute the processes.

I get that nodejs is single threaded, I have reviewed async but not figured out how this may apply given that callbacks play an integral role.

Essentially I want to call multiple ffmpeg processes in parallel without opening several terminal windows and handle callbacks such as errors or exit.

e.g.

//spawn ffmpeg with params
var ffmpegexec = spawn('ffmpeg', ['-i', 'pathname', '-vcodec', 'libx264', '-acodec', 'libfdk_aac', '-threads', '2', '-s', '320x240', 'filename.mp4');

//deal with output
ffmpegexec.stdout.on('data', function(data) {
    console.log("stdout: " + data);
});

ffmpegexec.stderr.on('data', function(data) {
    console.log("stderr : " + data);
});

ffmpegexec.on('exit', function(code) {

    console.log("exit: " + code);
});

I was thinking that perhaps i could launch each new process in a separate nodejs instance, or not depending on your recommendations

Answer 1

Just call spawn(...) multiple times consecutively.

Each call to process.spawn(...) starts a new OS process, a large number of which can be run concurrently. You will have to register separate stdout/stderr/exit handlers for each instance but if they are similar you can use the same (or similar) handlers.

For example, if you wanted to run the same commands on separate filenames you could do the following:

function runFfmpeg(filename) {
  var proc = process.spawn('ffmpeg', ['-i', 'pathname', ..., filename]);
  proc.stdout.on('data', function(data) { console.log("stdout: " + data); });
  proc.stderr.on('data', function(data) { console.log("stderr: " + data); });
  proc.on('exit', function(code) { console.log("exit: " + code); });
}

var filenames = ['foo.mp4', 'bar.mp4', 'gah.mp4'];
filenames.forEach(function(filename) { runFfmpeg(filename); });

Keep in mind that this could incur a huge load on your machine. You will likely need to throttle the number of threads/processes running at any time based on the resources available on the target physical machine (e.g. one process per physical CPU and one or two threads per CPU core).