Node JS workers - any need for them?

Question

Forgive my ignorance, but coming from a Django/Python background I can see the huge benefit of having a Celery queue working through the slower processes in the background whilst the web interface is updated as quick as possible.

However, with Node working asynchronously, is the use case for a queue system diminished hugely?

For example:

1 - a user posts something to the site, 2 - the site responds, then mails an administrator.

In Django, you'd send off the admin mail to a task, to be executed later, and then respond to the request. Celery sends the mail in the background.

In Node, you call your mailer, then respond to the request. The mailer then sends a callback to say DONE or not, by which point the user is already viewing the response.

So why would I use a queue with Node? I'm guessing when things are more complex than this - it seems that for trivial things like transactional mails, it's not necessary..

Or am I misunderstanding how it works!?

Answer 1

You're right, continuations are quite nice in node and if you're running everything in a single node process there is no immediate need for a queue. However, as node is single threaded, node won't be able to handle any new incoming requests while it's busy sending that email or processing that task (if it's a cpu intensive task)

So if your tasks take a while to process cpu wise, it might still be worthwhile using an external queue and a seperate process to handle those tasks/messages. If your tasks are io intensive and take a while because they are waiting for responses from other servers for example, then there's less of a need again as node deals well with io.

If you have a cpu intensive task, but you don't want to deploy a queue you could just create more node processes and machine instances and load balance across them, all letting them process these tasks. The disadvantage of this approach would be the you can't scale the website and background processing separately. (e.g. 5 instances dealing with web requests and a 2 worker instances)

Answer 2

No, there are always use cases for queues, even in the Node world. A basic approach that I've seen several people use to varying degrees of success is using a Redis backed queue to store messages or tasks. You may have one Node process adding items to the queue, with another Node instance processing items from the queue. In addition, take a look at the node modules list for queue modules, and you'll see a decent amount of implementations.