"Channel shutdown: connection error"

Question

I've set up on AWS a RabbitMQ cluster with two nodes, and enabled HA as described here. Then, I set up an Elastic Load Balancer mapping 5672 to the instances' 5672 port, with a periodic health check to the instances' 15672 port (the HTTP management port). Then, I started two listeners (one in each node), each one with 4 consumers, and pointed spring.rabbitmq.host at the load balancer's DNS. However, I'm periodically getting the following errors:

[E] [2015-03-14 23:13:23,890] [pool-4-thread-9         ] [CachingConnectionFactory        ] [                                    ] Channel shutdown: connection error
[E] [2015-03-14 23:13:23,891] [pool-4-thread-9         ] [CachingConnectionFactory        ] [                                    ] Channel shutdown: connection error
[E] [2015-03-14 23:13:23,891] [pool-4-thread-9         ] [CachingConnectionFactory        ] [                                    ] Channel shutdown: connection error
[E] [2015-03-14 23:13:23,891] [pool-4-thread-9         ] [CachingConnectionFactory        ] [                                    ] Channel shutdown: connection error
[W] [2015-03-14 23:13:24,758] [impleAsyncTaskExecutor-5] [SimpleMessageListenerContainer  ] [                                    ] Consumer raised exception, processing can restart if the connection factory supports it
com.rabbitmq.client.ShutdownSignalException: connection error
    at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:717) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:707) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:565) ~[amqp-client-3.4.2.jar!/:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
Caused by: java.io.EOFException
    at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290) ~[?:1.8.0_40]
    at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:139) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:534) ~[amqp-client-3.4.2.jar!/:?]
    ... 1 more
[I] [2015-03-14 23:13:24,761] [impleAsyncTaskExecutor-5] [SimpleMessageListenerContainer  ] [                                    ] Restarting Consumer: tags=[[amq.ctag-H-1ed6xO3GL7qW58bkLUZA]], channel=Cached Rabbit Channel: AMQChannel(amqp://[email protected]:5672qube,1), acknowledgeMode=AUTO local queue size=0
[W] [2015-03-14 23:13:24,762] [impleAsyncTaskExecutor-8] [SimpleMessageListenerContainer  ] [                                    ] Consumer raised exception, processing can restart if the connection factory supports it
com.rabbitmq.client.ShutdownSignalException: connection error
    at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:717) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:707) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:565) ~[amqp-client-3.4.2.jar!/:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
Caused by: java.io.EOFException
    at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290) ~[?:1.8.0_40]
    at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:139) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:534) ~[amqp-client-3.4.2.jar!/:?]
    ... 1 more
[I] [2015-03-14 23:13:24,762] [impleAsyncTaskExecutor-8] [SimpleMessageListenerContainer  ] [                                    ] Restarting Consumer: tags=[[amq.ctag-nJAmieDx-kSuxl9arsXiww]], channel=Cached Rabbit Channel: AMQChannel(amqp://[email protected]:5672qube,2), acknowledgeMode=AUTO local queue size=0
[W] [2015-03-14 23:13:24,767] [impleAsyncTaskExecutor-7] [SimpleMessageListenerContainer  ] [                                    ] Consumer raised exception, processing can restart if the connection factory supports it
com.rabbitmq.client.ShutdownSignalException: connection error
    at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:717) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:707) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:565) ~[amqp-client-3.4.2.jar!/:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
Caused by: java.io.EOFException
    at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290) ~[?:1.8.0_40]
    at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:139) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:534) ~[amqp-client-3.4.2.jar!/:?]
    ... 1 more
[I] [2015-03-14 23:13:24,768] [impleAsyncTaskExecutor-7] [SimpleMessageListenerContainer  ] [                                    ] Restarting Consumer: tags=[[amq.ctag-PAQvN8P57_9oiElyqYRJWw]], channel=Cached Rabbit Channel: AMQChannel(amqp://[email protected]:5672qube,3), acknowledgeMode=AUTO local queue size=0
[W] [2015-03-14 23:13:24,768] [impleAsyncTaskExecutor-6] [SimpleMessageListenerContainer  ] [                                    ] Consumer raised exception, processing can restart if the connection factory supports it
com.rabbitmq.client.ShutdownSignalException: connection error
    at com.rabbitmq.client.impl.AMQConnection.startShutdown(AMQConnection.java:717) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.AMQConnection.shutdown(AMQConnection.java:707) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:565) ~[amqp-client-3.4.2.jar!/:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
Caused by: java.io.EOFException
    at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290) ~[?:1.8.0_40]
    at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:95) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:139) ~[amqp-client-3.4.2.jar!/:?]
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:534) ~[amqp-client-3.4.2.jar!/:?]
    ... 1 more
[I] [2015-03-14 23:13:24,769] [impleAsyncTaskExecutor-6] [SimpleMessageListenerContainer  ] [                                    ] Restarting Consumer: tags=[[amq.ctag-w6apIlL78ViAnTOzx2Qejg]], channel=Cached Rabbit Channel: AMQChannel(amqp://[email protected]:5672qube,4), acknowledgeMode=AUTO local queue size=0

This does not happen if I point each consumer directly at one node (without using the load balancer). What can be causing this behaviout? How can I circumvent this?

Answer 1

Pinging the admin port will do nothing to keep the amqp connection(s) alive.

You need to configure heartbeats to a low enough value to prevent the load balancer from dropping an "idle" connection.

It looks like you might be using Spring Boot (based on the property name) and it doesn't currently expose requestedHeartbeats as a property; you will need to override the boot factory by wiring up your own rabbitConnectionFactory bean with the underlying rabbitmq connection factory heartbeat settings. See the documentation for details.

That said, there is no benefit gained from using a load balancer with Spring AMQP because it uses a single long-lived connection. Consider configuring your two instances in the addresses property instead and it will fail over to the second instance if the first is not available.