discovery.zen.minimum_master_nodes value for a cluster of two nodes

Question

I have two dedicated Windows Servers (Windows Server 2012R2, 128GB memory on each server) for ES (2.2.0). If I have one node on each server and the two nodes form a cluster. What is the proper value for

discovery.zen.minimum_master_nodes

I read this general rule in elasticsearch.yml:

Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):

I saw this SO thread:

Proper value of ES_HEAP_SIZE for a dedicated machine with two nodes in a cluster

There is an answer saying:

As described in Elasticsearch Pre-Flight Checklist, you can set discovery.zen.minimum_master_nodes to at least (N/2)+1 on clusters with N > 2 nodes.

Please note "N > 2". What is the proper value in my case?

Answer 1

N is the number of ES nodes (not physical machines but ES processes) that can be part of the cluster.

In your case, with one node on two machines, N = 2 (note that it was 4 here), so the formula N/2 + 1 yields 2, which means that both of your nodes MUST be eligible as master nodes if you want to prevent split brain situations.

If you set that value to 1 (which is the default value!) and you experience networking issues and both of your nodes can't see each other for a brief moment, each node will think it is alone in the cluster and both will elect themselves as master. You end up in a situation where you have two masters and that's not a good thing. Whereas if you set that value to 2 and you experience networking issues, the current master node will stay elected and the second node will never decide to elect itself as new master. Whenever network is back up, the second node will rejoin the cluster and continue serving requests.

The ideal topology is to have 3 dedicated master nodes (i.e. with master: true and data:false) and have discovery.zen.minimum_master_nodes set to 2. That way you'll never have to change the setting whatever the number of data nodes are part of your cluster.

So the N > 2 constraint should indeed be N >= 2, but I guess it was somehow implied, because otherwise you're creating a fertile ground for split brain situations.

Answer 2

Interestingly, in ES 7 discovery.zen.minimum_master_nodes is no longer need to be defined.

discovery.zen.minimum_master_nodes value for a cluster of two nodes https://www.elastic.co/blog/a-new-era-for-cluster-coordination-in-elasticsearch