RabbitMQ : Connection recovery mechanism

Question

I am using rabbit mq 3.4.1 java client library and not able to get the auto recovery mechanism work.

This is how I am creating the rabbit mq connection factory:

factory = new ConnectionFactory();
factory.setUsername(userName);
factory.setPassword(password);
factory.setVirtualHost(virtualHost);
factory.setAutomaticRecoveryEnabled(true);
factory.setNetworkRecoveryInterval(5);
factory.setRequestedHeartbeat(3);

After publishing of a message, if I shutdown the rabbit mq broker and bring it up again, I expect the recovery mechanism to kick in and have the connection restored to a 'sane' state. But I get the below error:

com.rabbitmq.client.AlreadyClosedException: connection is already closed due to connection error; protocol method: #method<connection.close>(reply-code=320, reply-text=CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', class-id=0, method-id=0)
    at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190) ~[amqp-client-3.4.1.jar:na]
    at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:291) ~[amqp-client-3.4.1.jar:na]
    at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:654) ~[amqp-client-3.4.1.jar:na]
    at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:631) ~[amqp-client-3.4.1.jar:na]
    at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:622) ~[amqp-client-3.4.1.jar:na]

Am I missing anything here? The only way to work around this problem is to register a ShutDownListener and re-initialize the rabbit mq connection factory, connection, and the channels.

Also to answer

"chrislott"

comment, I see the auto recovery kicking in to recover. I create a exchange by using a temporary channel:

Channel channel = connection.createChannel();
channel.exchangeDeclare(exchangeName, exchangeType, durable);
channel.close();

And I see the below exception when its trying to recover the topology:

Caught an exception when recovering topology Caught an exception while recovering exchange testSuccessfulInitVirtualHost_Exchange: channel is already closed due to clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)
com.rabbitmq.client.TopologyRecoveryException: Caught an exception while recovering exchange testSuccessfulInitVirtualHost_Exchange: channel is already closed due to clean channel shutdown; protocol method: #method<channel.close>(reply-code=200, reply-text=OK, class-id=0, method-id=0)
    at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverExchanges(AutorecoveringConnection.java:482)
    at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.recoverEntities(AutorecoveringConnection.java:467)
    at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.beginAutomaticRecovery(AutorecoveringConnection.java:411)
    at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.access$000(AutorecoveringConnection.java:52)
    at com.rabbitmq.client.impl.recovery.AutorecoveringConnection$1.shutdownCompleted(AutorecoveringConnection.java:351)
    at com.rabbitmq.client.impl.ShutdownNotifierComponent.notifyListeners(ShutdownNotifierComponent.java:75)
    at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:574)

The above exception is not seen if I do not close the channel that's used for creating the exchange.

Answer 1

My reading of the RabbitMQ ConnectionFactory#setAutomaticRecoveryEnabled(Boolean) method is that it primarily enables recovery from NETWORK failure.

Here's a nice discussion: https://www.rabbitmq.com/api-guide.html

For example, if your machine loses a route to the broker for a period of time, perhaps due to a switch or other failure, then the automatic recovery can re-establish a connection etc. The doc doesn't say anything about surviving broker shutdown/restart, I don't think your expectation is reasonable.

IMHO to recover from a broker restart, the shutdown-listener approach seems to be a solid approach.

Answer 2

Normally rabbit client should handle recovery itself - you shouldn't reimplement the same manually. At least try using lyra.

I had some problems during failover testing. Connections tend to start hanging forever on broker restart, so shutdown signal exception was the last thing in logs. I fixed it by setting:

factory.setConnectionTimeout(20000);

Also recovery didn't play well with temporary queues. If you have those you probably will have to do some additional handling (again, try lyra first).