Consuming a rabbitmq message queue with multiple threads (Python Kombu)

Question

I have a single RabbitMQ exchange with a single queue. I wish to create a daemon that runs multiple threads and works through this queue as quickly as possible.

The "work" involves communicating with external services, so there will be a fair amount of blocking going on within each consumer. As such, I want to have multiple threads all dealing with messages from the same queue.

I can achieve this by consuming the queue on my primary thread, and then farming the incoming work off to a pool of other threads, but is there a way to launch multiple consumers, each within their own threaded context?

Answer 1

Take a look at celery - it is designed to queue and process tasks from an AMPQ broker (but also works with other queuing backends). It handles multiprocess or multithreaded concurrency and makes it very easy to create and consume tasks.