Menu
I'd like to create a farm of processes that are able to OCR text. I've thought about using a single queue of messages which is read by multiple OCR processes.
I would like to ensure that:
Is that possible to do using AMQP?
I'm planning to use python and rabbitmq
451
A topic can be consumed by many consumer groups and each consumer group will have many consumers. A topic is divided into multiple partitions. A consumer in a consumer group is assigned to a partition. Only one consumer is assigned to a partition.
Support for multiple-consumer queues is a Message Queue feature (the JMS specification defines messaging behavior in the case of only one consumer accessing a queue). When multiple consumers access a queue, the load-balancing among them takes into account each consumer's capacity and message processing rate.
To run multiple consumers in the same group in one application, you will need to run each in its own thread. It is useful to wrap the consumer logic in its own object and then use Java's ExecutorService to start multiple threads each with its own consumer.
In RabbitMQ, we can have multiple consumers listening from one queue. This is good for fast procesing, but not so good for message ordering.
Yes, as @nailxx points out. The AMQP programming model is slightly different from JMS in that you only have queues, which can be shared between workers, or used privately by a single worker. You can also easily set up RabbitMQ to do PubSub use cases or what in JMS are called topics. Please go to our Getting Started page on the RabbitMQ web site to find a ton of helpful info about this.
Now, for your use case in particular, there are already plenty of tools available. One that people are using a lot, and that is well supported, is Celery. Here is a blog post about it, that I think will help you get started:
If you have any questions please email us or post to the rabbitmq-discuss mailing list.
97
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
