worker queue for nodejs?

Question

I am in the process of beginning to write a worker queue for node using node's cluster API and mongoose.

I noticed that a lot of libs exist that already do this but using redis and forking. Is there a good reason to fork versus using the cluster API?

edit and now i also find this: https://github.com/xk/node-threads-a-gogo -- too many options!

I would rather not add redis to the mix since I already use mongo. Also, my requirements are very loose, I would like persistence but could go without it for the first version.

Part two of the question: What are the most stable/used nodejs worker queue libs out there today?

Answer 1

Wanted to follow up on this. My solution ended up being a roll your own cluster impl where some of my cluster workers are dedicated job workers (ie they just have code to work on jobs).

I use agenda for job scheduling.

Cron type jobs are scheduled by the cluster master. The rest of the jobs are created in the non-worker clusters as they are needed. (verification emails etc)

Before that I was using kue but dropped it because the rest of my app uses mongodb and I didnt like having to use redis just for job scheduling.

Answer 2

Have u tried https://github.com/rvagg/node-worker-farm? It is very light weight and doesn't require a separate server.