ELASTICSEARCH: split brain

Question

Suppose we have two ElasticSearch servers A and B, both can become master. Assume also that A is master. Assume also that the minimum_master_nodes attribute is set to 1 as well.

If server A (the master) crashes for some reason, B becomes master. When A is up and running again, and joins the cluster, will it join again as master? I do not understand why this split brain could occur. If B becomes master, and then A rejoins the cluster, why would A become a master as well if B became the master?

Answer 1

At any time, there will only be one single master node in the cluster. If A leaves or crashes, B becomes the master. When A gets back up, it will simply join the cluster as a new master-eligible node and will wait until the node B crashes in order to become the new master.

The split brain situation can occur if the network between A and B breaks for a short moment. When that occurs, A and B are perfectly alive and think they are alone in the cluster, so since A doesn't see B anymore, A will elect itself as the master. At that point, you have two masters in your cluster, that's the split brain situation.

In order to prevent this, you should have an odd number of master-eligible nodes and make sure that minimum_master_nodes is set to 2 (number of master-eligible nodes / 2 + 1). When that's the case, it will only be possible to elect a new master if at least two master-eligible nodes are present and can communicate to each other in order to reach a quorum.

Running a cluster with two master-eligible nodes is calling for troubles, because if you set minimum_master_nodes: 1 you can end up in the split brain situation I described above. If you set minimum_master_nodes: 2, then if one node leaves or goes down, the cluster gets red and non-operational. That's why you should always run an odd number (bigger than one for obvious reasons) of master-eligible nodes.

Here's a great article with many more details on the split brain situation: http://blog.trifork.com/2013/10/24/how-to-avoid-the-split-brain-problem-in-elasticsearch/