Running multiple Laravel queue workers using Supervisor

Question

I using Laravel queues using a database driver and supervisor to keep a queue worker running all the time:

[program:laravel_queue]
command=php artisan queue:listen --timeout=1800 --tries=5
directory=/var/app/current
stdout_logfile=/var/app/support/logs/laravel-queue.log
logfile_maxbytes=0
logfile_backups=0
redirect_stderr=true
autostart=true
autorestart=true
startretries=86400
EOB

Some of the queue tasks can take around 10 minutes to complete.

I have 2 parts to the question:

1) How can i edit the above script to run multiple (e.g. 3) queue workers on the same queue.

2) Is there a way of scaling the number of queue workers running based on the number of jobs waiting to be processed?

The reason for question 2 is that we have batches of busy times and then lots of quiet times, so i don't really want to be wasting resources with 3 listeners running the whole time.

Answer 1

In supervisor config file. Add following code

process_name=%(program_name)s_%(process_num)02d
numprocs=8

This will create the 8 different processes of your program. You just need to change the value of numprocs=8 if you want.

You can check how many processes are running with sudo supervisorctl status

Answer 2

In supervisor you specify the amount of processes with the parameter numprocs, so you can add a line to your script that says:

numprocs=5

Now, you can do some clever things, for example if only some of the processes that runs on the queues takes too long, you can create a different set of queues processes to work those and other set for the light processes. To achieve that you can create a supervisor configuration with one queue name like --queue=longprocess and another one with --queue=lightprocess and in your program you dispatch the job in the corresponding queue, that way, long processes won't delay short processes.

You can also specify queue priorities within one supervisor configuration file such as --queue=lightprocess,longprocess. That way, your worker(s) will first look for lightprocess before running longprocess.

To answer your second question, no, in terms of supervisor all processes are running, it doesn't knows if the queue is busy or idle, therefore it can't kill processes and create them based on their use, so no, you can't have a dynamic configuration of creating more processes only when the ones you have are busy.

A note, if you assign more than 1 numprocs, you must add the number of process to the name. According to supervisor configuration docs:

Supervisor will start as many instances of this program as named by numprocs. Note that if numprocs > 1, the process_name expression must include %(process_num)s (or any other valid Python string expression that includes process_num) within it.

Supervisor configuration docs: http://supervisord.org/configuration.html