ZooKeeper - adding peers dynamically?

Question

I'm new to ZooKeeper. This is what I need.

I've a network of peers.

1. At t=t_1 -> [peer-1 (Leader), peer-2] peer-1 is the master and all clients connect to this node.

2. At t=t_2 -> [peer-1 (Leader), peer-2, peer-3] At some later time peer-3 joins the group. Is it possible to add peer-3 to the list of zookeeper servers "dynamically" ( i.e., without restarting ZooKeeper on peer-1 ) ?

3. At t=t_3 -> [peer-3 (Leader), peer-4] After a while both peer-1 and peer-2 leave the group (e.g., die or are switched off.) Assuming that there is a way to dynamically add peer-3 and peer-4 to the group peer-3 becomes the leader and all client requests are send to peer-3.

Are there any other options that I can use apart from using ZooKeeper to do something like this.

thanks.

Answer 1

At the moment, you can't dynamically change the configuration of a zookeeper cluster without restarting. There is an open issue to fix this, ZOOKEEPER-107. The paper describing the cluster membership algorithm is quite interesting, and can be found here.

You can change the configuration of the cluster by restarting server nodes 1 at a time. For example, if you cluster has servers A,B,C, and you want to replace server C with D, then you can do something like,

• Bring down C
• Bring up D, it's peer list is A,B,D
• Take down B
• Change B's peer list to A,B,D
• Bring up B
• Take down A Change A's peer list to A,B,D
• Bring up A
• Change the client configuration of all clients to point to A,B,D

At t=t_1, you have a cluster with 2 zookeeper nodes. This is quite brittle, as if either node goes down, you will not be able to establish quorum (floor(N / 2) + 1), and the cluster will be unavailable. Generally zookeeper clusters are odd numbers.

I'm not sure what you are trying to do when you say,

peer-3 becomes the leader and all client requests are send to peer-3.

You can't specify which node in a zookeeper cluster is the leader, the nodes themselves will elect their leader, and leadership will change as nodes go up and down. As well, clients typically don't always connect to the leader, but clients are given list of machines in the cluster, and connect randomly to one, reconnecting if the server they are connected to goes down. You can set the leaderServes option to specify that the leader does NOT server client connections.