RabbitMQ message consumers stop consuming messages

Question

Our team is in a spike sprint to choose between ActiveMQ or RabbitMQ. We made 2 little producer/consumer spikes sending an object message with an array of 16 strings, a timestamp, and 2 integers. The spikes are ok on our devs machines (messages are well consumed).

Then came the benchs. We first noticed that somtimes, on our machines, when we were sending a lot of messages the consumer was sometimes hanging. It was there, but the messsages were accumulating in the queue.

When we went on the bench plateform :

• cluster of 2 rabbitmq machines 4 cores/3.2Ghz, 4Gb RAM, load balanced by a VIP
• one to 6 consumers running on the rabbitmq machines, saving the messages in a mysql DB (same type of machine for the DB)
• 12 producers running on 12 AS machines (tomcat), attacked with jmeter running on another machine. The load is about 600 to 700 http request per second, on the servlets that produces the same load of RabbitMQ messages.

We noticed that sometimes, consumers hang (well, they are not blocked, but they dont consume messages anymore). We can see that because each consumer save around 100 msg/sec in database, so when one is stopping consumming, the overall messages saved per seconds in DB fall down with the same ratio (if let say 3 consumers stop, we fall around 600 msg/sec to 300 msg/sec).

During that time, the producers are ok, and still produce at the jmeter rate (around 600 msg/sec). The messages are in the queues and taken by the consumers still "alive".

We load all the servlets with the producers first, then launch all the consumers one by one, checking if the connexions are ok, then run jmeter.

We are sending messages to one direct exchange. All consumers are listening to one persistent queue bounded to the exchange.

That point is major for our choice. Have you seen this with rabbitmq, do you have an idea of what is going on ?

Thank you for your answers.

Answer 1

It's always worth setting the prefetch count when using basic.consume :

channel.basicQos(100);

before the channel.basicConsume line in order to ensure you never have more than 100 messages queued up in your QueueingConsumer.