Elasticsearch error not enough master nodes discovered during pinging (Although I am able to ping its IP)

Question

I am trying to set up an Elastic Search cluster with two nodes but I keep getting this error.

[o.e.d.z.ZenDiscovery     ] [master-node] not enough master nodes discovered ?during pinging (found [[Candidate{node={master-node}{EzeNCWZBR0aAKUP6Va9ZwA}{ZFq4O019Q2WVEWiISBQi2g}{ip-172-31-6-165}{172.31.6.165:9300}{ml.machine_memory=8369979392, xpack.installed=true, ml.max_open_jobs=20, 
ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again

Here are my node configuration:

1. Master Node
    cluster.name: prod-analytic-stats
     node.name: master-node
     path.data: /var/lib/elasticsearch
     path.logs: /var/log/elasticsearch
     network.host: ip-172-31-6-165
     discovery.zen.ping.unicast.hosts: ["ip-172-31-21-37", "ip-172-31-6-165"]
     discovery.zen.minimum_master_nodes: 2



2. data node
     cluster.name: prod-analytic-stats
     node.name: replica-node
     path.data: /var/lib/elasticsearch
     path.logs: /var/log/elasticsearch
     network.host: ip-172-31-21-37
     discovery.zen.ping.unicast.hosts: ["ip-172-31-21-37", "ip-172-31-6-165"]
     discovery.zen.minimum_master_nodes: 1

However I am able to ping data node from master node.

root@ip-172-31-6-165:/etc/elasticsearch# ping 172.31.21.37
PING 172.31.21.37 (172.31.21.37) 56(84) bytes of data.
64 bytes from 172.31.21.37: icmp_seq=1 ttl=255 time=1.18 ms
64 bytes from 172.31.21.37: icmp_seq=2 ttl=255 time=1.07 ms
64 bytes from 172.31.21.37: icmp_seq=3 ttl=255 time=1.14 ms
64 bytes from 172.31.21.37: icmp_seq=4 ttl=255 time=1.08 ms
64 bytes from 172.31.21.37: icmp_seq=5 ttl=255 time=1.16 ms
64 bytes from 172.31.21.37: icmp_seq=6 ttl=255 time=1.12 ms
64 bytes from 172.31.21.37: icmp_seq=7 ttl=255 time=1.21 ms
64 bytes from 172.31.21.37: icmp_seq=8 ttl=255 time=1.05 ms
--- 172.31.21.37 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7008ms
rtt min/avg/max/mdev = 1.051/1.131/1.219/0.068 ms

Answer 1

discovery.zen.minimum_master_nodes must have the same value on all nodes. If you set it to 2 on both nodes, then both nodes must be eligible masters, i.e. node.master: true.

If you set it to 1, then only a single node needs to have node.master: true.

In your case, you have minimum_master_nodes: 1 on your data node ip-172-31-21-37 and minimum_master_nodes: 2 on your master node ip-172-31-6-165, so it's not going to work.