Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Delayed_job - Multiple parallel queues?

I am using delayed_job and moved to a new beefier server. So now I would like to run parallel jobs, as now I have the POWER!, but am confused on whether delayed_job can run multiple parallel queues?

This question suggested that there are named queues, but do these all run off the one table and are thus sequential?

At the bottom @Jesse Wolgamott suggests that you can create a table for each queue that will then run in parrallel.

Has anyone done this and can they point me to how it is done?

like image 513
slotishtype Avatar asked Feb 28 '12 09:02

slotishtype


2 Answers

It is possible and I am doing it all the time. In our case we need multiple jobs for processing three different kinds of jobs say queue_a, queue_b, and queue_c. The system will make entries in the delayed_job table and the queue named appropriately.

Start multiple delayed jobs like

RAILS_ENV=production script/delayed_job -i first --queue=queue_a start
RAILS_ENV=production script/delayed_job -i second --queue=queue_a start
RAILS_ENV=production script/delayed_job -i third --queue=queue_b  start
RAILS_ENV=production script/delayed_job -i fourth --queue=queue_c start

Each command will create a delayed_job, so there will be now 4 parallel jobs, two of them serving queue_a and one each for queue_b and queue_c. The key thing here is the identifier that is passed through the -i option which specifies the instance name, and we can start and stop jobs as required.

Another option is to use worker pools.

RAILS_ENV=production script/delayed_job --pool=tracking --pool=mailers,tasks:2 --pool=*:2 start

That command will start 1 worker for the tracking queue, 2 workers for the mailers and tasks queues, and 2 workers for any jobs.

like image 110
Ggmon Avatar answered Oct 23 '22 00:10

Ggmon


With bundler in production:

RAILS_ENV=production bundle exec script/delayed_job -n 4 start

or without bundler

ruby script/delayed_job -n 4 start

like image 40
slotishtype Avatar answered Oct 23 '22 00:10

slotishtype