Restoring backed up queued messages in RabbitMQ not working

Question

According to the following posts:

http://rabbitmq.1065348.n5.nabble.com/RabbitMQ-Backup-td18268.html

http://rabbitmq.1065348.n5.nabble.com/rabbitmq-server-Mnesia-backup-and-restore-td28598.html

It is possible to backup then restore durable queued messages by performing the following steps:

To back up we have to:

1- stop the rabbitmq server: # rabbitmqctl stop_app

2- copy (tar) the folder "/var/lib/rabbitmq/mnesia/": # tar -cvf mnesia.tar /var/lib/rabbitmq/mnesia/

3- start the rabbitmq server: # rabbitmqctl start_app

Then to restore them we have to:

1- stop the rabbitmq server: # rabbitmqctl stop_app

2- copy back (untar) the folder "/var/lib/rabbitmq/mnesia/": # tar -xvf mnesia.tar -C /

3- start the rabbitmq server: # rabbitmqctl start_app

But when trying to apply those steps on a rabbitmq cluster or even on a single node, could not restore any message.

Have also noticed that the content of: /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-node1/msg_store_transient where seems to be stored the queued messages, is always cleaned just after rabbitmq server is restarted (stop_app and start_app). Then have tried to copy the backed up tar, after starting rabbitmq, to not get the folder /msg_store_transient cleaned, but this didn’t help either (IOW no sign of restored messages in the web management console).

We are performing our tests on virtual machines with: Ubuntu-14.04, Erlang-R16B03, RabbitMQ-3.4.1, and with durable queues created by a java client.

Will appreciate any help or tip to properly restore queued message especially after a rabbitmq server failure.

Answer 1

Persistency needed

Published messages should be marked as persistent - by supplying a delivery_mode property with a value 2. See https://www.rabbitmq.com/tutorials/tutorial-two-python.html

In Java client we can set that as follows:

AMQP.BasicProperties basicProperties = new AMQP.BasicProperties().builder().deliveryMode(2).build();

Channel channel = initializeChannel(...);

channel.basicPublish(exchange, rootinKey, basicProperties, message body in bytes);

Storage Location

The published messages were stored in

/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-node1/msg_store_persistent

directory, instead of

/var/lib/rabbitmq/mnesia/rabbit@rabbitmq-node1/msg_store_transient

And the content of mnesia/rabbit@rabbitmq-node1/msg_store_persistent** was not cleaned up after rabbitmq restarted.

Also have constated that after a restart of a single node or all cluster's nodes, all messages marked as persistent were available.

Restore for single node configurations

Restoring backed up messages also worked out properly in a single node rabbitmq configuration, for that have backed up only /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-node1 dir which was fine and enough to restore the queue with all its messages, after have deleted the queue with its content using rabbitmq manager web console.

Restore for cluster configurations and changed node names

But in a cluster of multiple nodes have failed to restore backed up messages after have deleted the queue from all cluster nodes.

As a workaround have tried to restore a backup from a cluster to a single node rabbitmq server with the same node name (in my case the node name is rabbit@rabbitmq-node1) and succeeded by performing the following steps:

1. Adding a durable queue with same name like the one which was backed up, this should create the same directory (with same code-name) in **mnesia\rabbit@rabbitmq-node1\queues** directory.

2. Stopping rabbitmq and replacing only the following directories from the backup:

   /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-node1/msg_store_persistent

   /var/lib/rabbitmq/mnesia/rabbit@rabbitmq-node1/queues

3. Starting rabbitmq, if the Queued messages do not appear in the Overview of the management web console, then restart the server using:

   \# /etc/init.d/rabbitmq-server restart

This worked for me to restore (IOW migrate) queued messages of a particular queue from a Cluster to a single node rabbitmq server.