When to use multiple supervisors in Laravel Horizon

Question

In my config/horizon I now have:

'production' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['arbitrage'],
                'balance' => 'auto',
                'processes' => 2,
                'tries' => 1,
            ],
            'supervisor-2' => [
                'connection' => 'redis',
                'queue' => ['trade'],
                'balance' => 'auto',
                'processes' => 4,
                'tries' => 1,
            ],
            'supervisor-3' => [
                'connection' => 'redis',
                'queue' => ['balance', 'trade_meta'],
                'balance' => 'auto',
                'processes' => 5,
                'tries' => 1,
            ],
            'supervisor-4' => [
                'connection' => 'redis',
                'queue' => ['notifications'],
                'balance' => 'auto',
                'processes' => 2,
                'tries' => 1,
            ],
        ],

I've done this because I want to make sure that each of those queues get their assigned workers, that the different queues are running in parallel and that they don't impact each other, even if, for instance, the notifications queue has dozens of jobs and the trade queue is empty.

My question is if this makes sense, when exactly should I use multiple supervisors? Or: how would my above config differ from doing:

    'supervisor-1' => [
        'connection' => 'redis',
        'queue' => ['arbitrage', 'trade', 'balance', 'trade_meta', 'notifications'],
        'balance' => 'auto',
        'processes' => 10,
        'tries' => 1,
    ],

Would this only matter if there more than 10 jobs at the same time on these queues combined given that I have specified 10 workers?

There could be about 10 to 30 jobs at the same time on the queue, and this can grow in the future. That's the reason why I build my config this way but I'm not sure what defining multiple supervisors exactly means.

Answer 1

I would do it in situations where I need to make sure there are workers assigned to more than one queue at the same time, or if I needed a different connection or balancing method for some of my queues.

In your first example, if you only have jobs in the notifications queue, all workers on the other queues would be idle. This means that a large amount of jobs in the arbitrage queue, would not hold up sending of notifications.

In your latter example, you would be sure that all processes would be in use as long as there are enough jobs to feed them overall, however a large queue in either queue, would delay execution of jobs in the queues listed after it.