Menu
How do you use a Celery queue with the same name for multiple apps?
I have an application with N client databases, which all require Celery task processing on a specific queue M.
For each client database, I have a separate celery worker that I launch like:
celery worker -A client1 -n client1@%h -P solo -Q long
celery worker -A client2 -n client2@%h -P solo -Q long
celery worker -A client3 -n client3@%h -P solo -Q long
When I ran all the workers at once, and tried to kick off a task to client1, I found it never seemed to execute. Then I killed all workers except for the first, and now the first worker receives and executes the task. It turned out that even though each worker's app used a different BROKER_URL, using the same queue caused them to steal each others tasks.
This surprised me, because if I don't specify -Q, meaning Celery pulls from the "default" queue, this doesn't happen.
How do I prevent this with my custom queue? Is the only solution to include a client ID in the queue name? Or is there a more "proper" solution?
539
Celery requires a solution to send and receive messages; usually, this comes in the form of a separate service called a message broker. In celery, the broker is Redis, RabbitMQ, etc who conveying the message between a client and celery.
celery beats only trigger those 1000 tasks (by the crontab schedule), not run them. If you want to run 1000 tasks in parallel, you should have enough celery workers available to run those tasks.
Not only CAN Celery run more than one worker, that is in fact the very point, and reason Celery even exists and it's whole job is to manage not just multiple workers but conceivably across machines and distributed.
Celery: Distributed task queue. Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well; RabbitMQ: A messaging broker - an intermediary for messaging.
For multiple applications I use different Redis databases like
redis://localhost:6379/0
redis://localhost:6379/1
etc.
52
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
