RabbitMQ queue design and scaling

Question

In my application I have a queue that could potentially become very big. What if I discover that there's no more space on my machine? How can I split my queue on multiple machines? Maybe RabbitMQ philosophy is different and I should create multiple queues instead of one big queue..?

Best, Flavio

Answer 1

As you can read on RabbitMQ mailing list thread http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-scalability-design-question-td28323.html, the solution I came up with is to implement the competing consumers pattern (many consumers on a queue) that when detects a special message (with stop flag on) sends a STOP message to a topic exchange.

This stop message is received by a "master" consumer for that queue that starts polling a Zookeeper (via curator) until all children of a ceratain zkNode have been deleted (using Zookpeer as a registry of queue-consumers in this case). When all consumers have finished their stopping phase the "master" consumer does some task and re-enable the original queue consumers sending a RESTART message to the topic exchange (where each consumer is listening to with a dedicated queue).

I hope this could help (or "inspire") someone else..

Answer 2

RabbimMQ provides clustering and high availability features right out of the box, you just have to configure them to your needs.

Actually, AMQP can hold messages of any size, but in most cases messages are just up to 32Kb in 99%, I guess. You can calculate estimate resource usage (min/max/avg) and make further decision do cluster or not to cluster.