Spring AMQP - concurrent publishing to RabbitMQ Broker seems to block

Question

I have a little spring boot app that acts as a producer to some queues. It uses the following dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-amqp</artifactId>
    <version>1.4.0.RELEASE</version>
</dependency>

Within the app I start 4 threads. Within each task random messages get created with a maximum size of 10KB. Those messages are sent to a direct exchange using the method convertAndSend from AmqpTemplate. For each thread a separate channel is used. So far everything is running smooth.

During runtime I had a look into the JVM using Java VisualVM. I could see the 4 threads working but was a bit surprised, that the threads are most of the time in monitor (blocked) state. See the following screenshot:

http://i.stack.imgur.com/i8uX7.png

And here is one piece of a stack trace:

java.lang.Thread.State: BLOCKED
at com.rabbitmq.client.impl.SocketFrameHandler.writeFrame(SocketFrameHandler.java:144)
- waiting to lock <1407a47> (a java.io.DataOutputStream) owned by "Thread-7" t@35
at com.rabbitmq.client.impl.AMQConnection.writeFrame(AMQConnection.java:514)
at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:102)
- locked <316a68> (a com.rabbitmq.client.impl.CommandAssembler)
at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:334)
- locked <1d8e172> (a java.lang.Object)
at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:310)
- locked <1d8e172> (a java.lang.Object)
at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:657)

So that doesn't look like parallel publishing to me.

Can anyone tell if that behavior is correct? To me it seems that it's not, but I can't figure out whats going wrong.

If I start the same producer app on an additional machine, the blocking state seems to get worse, as if the rabbitmq broker is synchronizing the access. The broker itself is fine. No physical limits hit. No flow control. Even the incoming message rates get almost doubled with an additional producer app.

That does't make any sense to me. Does anyone have an idea? Do I have to worry?

Thanks in advance!

Answer 1

Probably this is the problem:

https://groups.google.com/forum/#!topic/rabbitmq-users/15cv2qroCps

I also faced it, this feature is the reason:

The RabbitMQ server detects the total amount of RAM installed in the computer on startup and when rabbitmqctl set_vm_memory_high_watermark fraction is executed. By default, when the RabbitMQ server uses above 40% of the installed RAM, it raises a memory alarm and blocks all connections that are publishing messages.

So you can fix it if you increase this threshold / add plus memory to the machine.

http://www.rabbitmq.com/memory.html