Celery: number of workers vs concurrency

Question

What is the difference between having:

1. one worker with concurrency 4 or
2. two workers with concurrency 2 each

for the same queue.

Thanks

Answer 1

I assume that you are running both workers in the same machine. In that case I would recommend you to maintain one worker for a queue.

• Two workers for the same queue does not benefit you by any means. It would just increase the memory wastage.
• Two or more workers when you have multiple queues, to maintain priority or to allocate different number of cores to each worker.
• Two or more workers for a single queue is useful if you run the workers in different machines. The workers in different machines consumes tasks from same queue, you could allocate concurrency based on the cores available in each machine.

I do realise I responded 2+ years later. But I just thought I'll put it here for anyone who still has similar doubts.