Menu
I'm using a RabbitMQ broker, and there is a Celery worker which is subscribed to the broker. From my testing, it looks like RabbitMQ treats messages in FIFO order. Because one queue has been populated, then another, then another, and so on, my worker consumes all the messages from queue 1, and only moves on to queue 2 once it is done with queue 1.
Is it possible to change this behavior? I would like the Celery worker to consume in a round-robin style instead, ie consume a message from queue 1, then a message from queue 2, and so on, only coming back to queue 1 once a message has been consumed from each of the other queues.
264
By default, RabbitMQ will send each message to the next consumer, in sequence. On average every consumer will get the same number of messages. This way of distributing messages is called round-robin.
Celery requires a solution to send and receive messages; usually, this comes in the form of a separate service called a message broker. In celery, the broker is Redis, RabbitMQ, etc who conveying the message between a client and celery.
celery -A merlin worker --concurrency 2. The --prefetch-multiplier argument sets how many tasks are requested from the task server per worker thread. If --concurrency is 2 and --prefetch-multiplier is 3, then 6 tasks will be requested from the task server by the worker threads.
Celery: Distributed task queue. Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well; RabbitMQ: A messaging broker - an intermediary for messaging.
Yes, you have to reduce your prefetch_count to 1 so only 1 message is fetched at a time. In Celery you can archive this by setting CELERYD_PREFETCH_MULTIPLIER to 1. You may also want to set task_acks_late = True, make sure you read the documentation on both.
from the Celery docs:
To disable prefetching, set worker_prefetch_multiplier to 1. Changing that setting to 0 will allow the worker to keep consuming as many messages as it wants.
113
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
