How do you backup Zookeeper?

Question

I have read that copying the data directory will work. But, that is a combination of logs and snapshots. How do folks backup a zookeeper infrastructure ? Export ? Dump ? Custom script ? What are the best practices ?

Answer 1

Zookeeper writes a snapshot once it determines that it has enough transactions and every new snapshot completely supersedes older ones. So the latest snapshot + the transaction log from the time of the snapshot is enough to recover to current state. To make the calculations easier, you can simply backup the last 3 snapshots(in case of corruption of the latest snap) and the transaction logs from the timestamp corresponding to the earliest snapshot. The links below have some more details.

1. http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_dataFileManagement
2. http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance

Answer 2

There's a very nice tool called zk-shell that can do an enormous amount of things with Zookeeper. It has a mirror command that can copy an entire Zookeeper tree recursively to/from Zookeeper or local JSON file.

Source & documentation: https://github.com/rgs1/zk_shell

Installation on Centos 7:

yum install python2-pip
pip install zk_shell

Example to back up a zookeeper tree to a local JSON file /tmp/zookeeper-backup.json:

zk-shell localhost:2181 --run-once 'mirror / json://!tmp!zookeeper-backup.json/'

Answer 3

Netflix provided a solution for this called exhibitor. It's a "ZooKeeper co-process for instance monitoring, backup/recovery, cleanup and visualization."