failed to send join request to master elastic search 5.4 cluster

Question

I have configured elastic search with 3 nodes My cluster is not able to find the master node

Logs of elastic search

[2018-02-24T02:39:39,106][INFO ][o.e.d.z.ZenDiscovery     ] [node3] failed to send join request to master [{node1}{MO28S_KZQaih1pd-ERrvIA}{SBwZDgFcTz2xTLOyh2fZAA}{192.168.2.xxx}{192.168.2.xxx:9300}], reason [RemoteTransportException[[node1][192.168.2.xxx:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node1}{MO28S_KZQaih1pd-ERrvIA}{SBwZDgFcTz2xTLOyh2fZAA}{192.168.2.xxx}{192.168.2.xxx:9300}] not master for join request]; ], tried [3] times
[2018-02-24T02:39:42,332][INFO ][o.e.d.z.ZenDiscovery     ] [node3] failed to send join request to master [{node1}{MO28S_KZQaih1pd-ERrvIA}{SBwZDgFcTz2xTLOyh2fZAA}{192.168.2.xxx}{192.168.2.xxx:9300}], reason [RemoteTransportException[[node1][192.168.2.xxx:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node1}{MO28S_KZQaih1pd-ERrvIA}{SBwZDgFcTz2xTLOyh2fZAA}{192.168.2.xxx}{192.168.2.xxx:9300}] not master for join request]; ], tried [3] times

My elasticsearch.yml file

cluster.name: cluster-testing 
node.name: node3
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.2.xxx
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.2.xxx", "192.168.2.xxx", 
                                    "192.168.2.xxx"]
discovery.zen.minimum_master_nodes: 2 
gateway.recover_after_nodes: 2 
gateway.expected_nodes: 3

I am not able to find where is the issue

Answer 1

I faced a similar error, but my case was a little different.

I was trying to boot up 2 ES nodes on the same machine. I had copied my ES directory to start a new node and I was getting RemoteTransportException this exception.

After Googling a little bit I found out that I had to delete nodes directory in the copied directory to boot that node. This directory is usually present in {ES_HOME}/data/nodes or /var/lib/elasticsearch/nodes.

Source: https://github.com/elastic/elasticsearch/issues/21405

Answer 2

To turn a node into a master node you need to add

node.master: true

into elasticsearch.yml.

From your configuration

discovery.zen.minimum_master_nodes: 2 

at least two of your nodes must set node.master to true

EDIT:

you can find below working configuration for our project (3*masters and 6*slaves):

MASTER:

cluster.name: VAL_elasticsearch
node.name: MASTER-${HOSTNAME}
node.data: false
node.master: true
path.data: /data
path.logs: /local/opt/logs
path.repo: ["/home/data_ElasticSearchBackup"]
bootstrap.memory_lock: true
bootstrap.seccomp: false
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["ip_master1","ip_master2","ip_masterx"]
discovery.zen.minimum_master_nodes: 2

SLAVE:

cluster.name: VAL_elasticsearch
node.name: DATA-${HOSTNAME}
node.data: true
node.master: false
path.data: /data
path.logs: /local/opt/logs
path.repo: ["/home/data_ElasticSearchBackup"]
bootstrap.memory_lock: true
bootstrap.seccomp: false
network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["ip_master1","ip_master2","ip_masterx"]
discovery.zen.minimum_master_nodes: 2

Are you sure all ports are open between all elastic instances ?