Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

discovery.zen.minimum_master_nodes value for a cluster of two nodes

I have two dedicated Windows Servers (Windows Server 2012R2, 128GB memory on each server) for ES (2.2.0). If I have one node on each server and the two nodes form a cluster. What is the proper value for

discovery.zen.minimum_master_nodes

I read this general rule in elasticsearch.yml:

Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):

I saw this SO thread:

Proper value of ES_HEAP_SIZE for a dedicated machine with two nodes in a cluster

There is an answer saying:

As described in Elasticsearch Pre-Flight Checklist, you can set discovery.zen.minimum_master_nodes to at least (N/2)+1 on clusters with N > 2 nodes.

Please note "N > 2". What is the proper value in my case?

like image 849
curious1 Avatar asked Feb 29 '16 03:02

curious1


2 Answers

N is the number of ES nodes (not physical machines but ES processes) that can be part of the cluster.

In your case, with one node on two machines, N = 2 (note that it was 4 here), so the formula N/2 + 1 yields 2, which means that both of your nodes MUST be eligible as master nodes if you want to prevent split brain situations.

If you set that value to 1 (which is the default value!) and you experience networking issues and both of your nodes can't see each other for a brief moment, each node will think it is alone in the cluster and both will elect themselves as master. You end up in a situation where you have two masters and that's not a good thing. Whereas if you set that value to 2 and you experience networking issues, the current master node will stay elected and the second node will never decide to elect itself as new master. Whenever network is back up, the second node will rejoin the cluster and continue serving requests.

The ideal topology is to have 3 dedicated master nodes (i.e. with master: true and data:false) and have discovery.zen.minimum_master_nodes set to 2. That way you'll never have to change the setting whatever the number of data nodes are part of your cluster.

So the N > 2 constraint should indeed be N >= 2, but I guess it was somehow implied, because otherwise you're creating a fertile ground for split brain situations.

like image 153
Val Avatar answered Sep 20 '22 01:09

Val


Interestingly, in ES 7 discovery.zen.minimum_master_nodes is no longer need to be defined.

discovery.zen.minimum_master_nodes value for a cluster of two nodes https://www.elastic.co/blog/a-new-era-for-cluster-coordination-in-elasticsearch

like image 31
Anurag Sharma Avatar answered Sep 18 '22 01:09

Anurag Sharma