Can Zookeeper remain highly available if one of three nodes fails

Question

If I have three zookeeper nodes, one leader and two followers. What happens if the leader dies? Would the new leader be selected from the remaining two? If so, what happens if the new leader also dies? Would the last one still serve clients?

Answer 1

In a ZooKeeper cluster, if a leader dies the other remaining nodes will elect a leader among themselves. This will usually be very quick and there should be no visible outage to the clients. The clients that were connected to the leader that died will reconnect to one of the other nodes.

As for the second question - in a 3 node cluster, if 2 of the nodes die, the third one will not be serving requests. The reason for that is that the one remaining node cannot know if it is in fact the only survivor or if it has been partitioned off from others. Continuing to serve request at that point could cause a split brain scenario and violate the core ZooKeeper guarantee.