NodeJS batch multi processing - child processes in a pool (or multithreading)

Question

NodeJS batch multi threading processing - child processes in a pool.

I know a child process is a process, not a thread. I used wrong semantics, because most people know what your intent is when you speak of "multithreading". So I'll keep it in the title.

---

Imagine a scenario where you continuously have multiple similar and complex things to do using a single custom function or module. It makes a lot of sense to use all your available cores/threads (e.g. 8/16), which is what child_process.fork() is for.

Ideally, you are going to want a number of simultaneous workers and send/callback messages to/from one controller.

node-cpool, fork-pool, child-pool are some modules that do exactly this, but they seem old/unmaintained/impopular.

There are a ton of similar-ish modules, but these seem the most relevant. What they all have in common is a couple of commits, hardly starred, hardly forked, and abandoned.

What is usually the case when I can't find something for a task that seems like something that makes sense in every way, is that there is an even better way that I am missing. Hence my question.

How do I have a managed, queued, multithreaded pool of parallel fork()s for my custom module that does some CPU intensive work?

Multithreaded modules like TAGG and webworker-threads are not the same because they don't support full modules (with binary compiled components).

---

PS

I am now using fork-pool which seems to do exactly what I want, with some quirks, but I can't believe that such an unknown and impopular module would be the only viable option here.

Answer 1

I would suggest using something like Redis as your Queue. Here's a tutorial of creating a message bus in Node with Redis and Kue. This will scale pretty well and allow you to have multiple processes, threads, or even machines producing and consuming items to/from the queue.