How do you handle recovering from a faulty connection using RabbitMQ java client library?

Question

I'm interested in knowing how other people handle recovering from a faulty connection using the official RabbitMQ java client library. We are using it to connect our application servers to our RabbitMQ cluster and we have implemented a few different ways to recover from a connection failure, but non of them feel quite right.

Imagine this pseudo application:

public class OurClassThatStartsConsumers {
    Connection conn;

    public void start() {
        ConnectionFactory factory = new ConnectionFactory();
        factory.setUsername("someusername");
        factory.setPassword("somepassword");
        factory.setHost("somehost");
        conn = factory.newConnection();

        new Thread(new Consumer(conn.createChannel())).start();
    }
  }

class Consumer1 implements Runnable {
    public Consumer1(Channel channel) {
         this.channel = channel;
    }

    @Override
    public void run() {
        while (true) {
             ... consume incoming messages on the channel...
            // How do we handle that the connection dies?
        }
    }
}

In the real world we have several hundreds of consumers. So what happens if the connection dies? In the above example Consumer1 can not recover, when the connection closes, the Channel also closes, a state from which we can not recover. So lets look at some ways to solve this:

Solution A)

Let every consumer have their own connection and register the events that trigger when the connection dies and then handle reconnecting.

Pros: It works

Cons:

• Since we have a lot of consumers, we probably do not want that many connections.
• We might possibly have a lot of duplicated code for reconnecting to rabbit and handle reconnecting

Solution B)

Have each consumer use the same connection and subscribe to it's connection failure events.

Pros: Less connections than in Solution A

Cons: Since the connection is closed we need to reopen/replace it. The java client library doesn't seem to provide a way to reopen the connection, so we would have to replace it with a new connection and then somehow notify all the consumers about this new connection and they would have to recreate the channels and the consumers. Once again, a lot of logic that I don't want to see in the consumer ends up there.

Solution C)

Wrap Connection and Channel classes is classes that handle the re-connection logic, the consumer only needs to know about the WrappedChannel class. On a connection failure the WrappedConnection will deal with re-establishing the connection and once connected the WrappedConnection will automatically create new Channels and register consumers.

Pros: It works - this is actually the solution we are using today.

Cons: It feels like a hack, I think this is something that should be handled more elegantly by the underlying library.

Maybe there is a much better way? The API documentation does not talk that much about recovering from a faulty connection. Any input is appreciated :)

Answer 1

Since version 3.3.0 you can use automatic recovery, which is a new feature of the Java client. From the Java API guide (http://www.rabbitmq.com/api-guide.html#recovery)

To enable automatic connection recovery, use factory.setAutomaticRecovery(true):

Answer 2

Got some good answers on the RabbitMQ mailing list, basically suggesting solution C as i listed above.

Solution C)

Wrap Connection and Channel classes is classes that handle the re-connection logic, the consumer only needs to know about the WrappedChannel class. On a connection failure the WrappedConnection with deal with re-establishing the connection and once connected the WrappedConnection will automatically create new Channels and register consumers.

Pros: It work - this is actually the solution we are using today.

Cons: It feels like a hack, I think this is something that should be handled more elegantly by the underlying library.

This is what the two clients built on top of the Java one — Langohr and March Hare — do. It’s not a hack but a necessary work around because connection recovery is currently not performed by the Java client (it should be a core feature, if you ask me).

So this is a viable approach.

Take a look at Lyra, too: https://github.com/jhalterman/lyra.

MK

Software Engineer, Pivotal/RabbitMQ

And:

Hi Peter,

Solution C is actually pretty reasonable. There's not much to gain from using multiple connections to the same server if you're trying to guard against network failures or cluster partitions. If one connection dies, they all likely will. Wrapping and recovering connections/channels works fine, and as Michael mentioned, you might also check out Lyra since it handles the various corner cases involved in recovering resources for you.

Cheers, Jonathan

Read the full thread here:

http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-October/031564.html

http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2013-November/031573.html