Multiple consumers & producers connected to a message queue, Is that possible in AMQP?

Question

I'd like to create a farm of processes that are able to OCR text. I've thought about using a single queue of messages which is read by multiple OCR processes.

I would like to ensure that:

• each message in queue is eventually processed
• the work is more or less equally distributed
• an image will be parsed only by one OCR process
• An OCR process won't get multiple messages at once (so that any other free OCR process can handle the message).

Is that possible to do using AMQP?

I'm planning to use python and rabbitmq

Answer 1

Yes, as @nailxx points out. The AMQP programming model is slightly different from JMS in that you only have queues, which can be shared between workers, or used privately by a single worker. You can also easily set up RabbitMQ to do PubSub use cases or what in JMS are called topics. Please go to our Getting Started page on the RabbitMQ web site to find a ton of helpful info about this.

Now, for your use case in particular, there are already plenty of tools available. One that people are using a lot, and that is well supported, is Celery. Here is a blog post about it, that I think will help you get started:

If you have any questions please email us or post to the rabbitmq-discuss mailing list.