How to set up two machines for a cluster with two nodes on each machine

Question

I have two dedicated machines for ES (2.2.0). The two machines have the same specs. Each runs on a Windows Server 2012 R2 and has 128GB memory. Regarding ES, I plan to have TWO nodes on each machine for the cluster.

I am looking at elasticsearch.yml to see how to configure each node to form a cluster.

The two machines on the same network with the following server names and IP addresses:

SRC01, 172.21.0.21
SRC02, 172.21.0.22

I am looking at elasticsearch.yml and I am not sure how to set things up. I guess that I need set proper values for Network and Discovery sections in elasticsearch.yml:

# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
# network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
# http.port: 9200
#
# --------------------------------- Discovery ----------------------------------
#
# Elasticsearch nodes will find each other via unicast, by default.
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
# discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
# discovery.zen.minimum_master_nodes: 3
#

I googled on the net and SO and hoped to find a complete config example for me to start, but failed to find one.

Any input or pointer is really appreciated.

UPDATE

With Val's help, here is the minimum elasticsearch.yml that I have on the four nodes (2 one each machine) after tests:

#----------SRC01, node 1---------
cluster.name: elastic
node.name: elastic_src01_1
network.host: 172.21.0.21
discovery.zen.ping.unicast.hosts: ["172.21.0.21","172.21.0.22"]


#----------SRC01, node 2---------
cluster.name: elastic
node.name: elastic_src01_2
network.host: 172.21.0.21
discovery.zen.ping.unicast.hosts: ["172.21.0.21","172.21.0.22"]


#----------SRC02, node 1---------
cluster.name: elastic
node.name: elastic_src02_1
network.host: 172.21.0.22
discovery.zen.ping.unicast.hosts: ["172.21.0.21","172.21.0.22"]


#----------SRC02, node 2---------
cluster.name: elastic
node.name: elastic_src02_2
network.host: 172.21.0.22
discovery.zen.ping.unicast.hosts: ["172.21.0.21","172.21.0.22"]

Here is the questions I got:

1. I started Node elastic_src01_1 and then Node elastic_src01_2 and they are on the same machine. When starting elastic_src01_2, I am able to see ES-generated messages the following (detected_master)

Logs excerpt:

[2016-02-28 12:38:33,155][INFO ][node                     ] [elastic_src01_2] version[2.2.0], pid[4620], build[8ff36d1/2016-01-27T13:32:39Z]
[2016-02-28 12:38:33,155][INFO ][node                     ] [elastic_src01_2] initializing ...
[2016-02-28 12:38:33,546][INFO ][plugins                  ] [elastic_src01_2] modules [lang-expression, lang-groovy], plugins [], sites []
[2016-02-28 12:38:33,562][INFO ][env                      ] [elastic_src01_2] using [1] data paths, mounts [[Data (E:)]], net usable_space [241.7gb],
net total_space [249.9gb], spins? [unknown], types [NTFS]
[2016-02-28 12:38:33,562][INFO ][env                      ] [elastic_src01_2] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-02-28 12:38:35,077][INFO ][node                     ] [elastic_src01_2] initialized
[2016-02-28 12:38:35,077][INFO ][node                     ] [elastic_src01_2] starting ...
[2016-02-28 12:38:35,218][INFO ][transport                ] [elastic_src01_2] publish_address {172.21.0.21:9302}, bound_addresses {172.21.0.21:9302}
[2016-02-28 12:38:35,218][INFO ][discovery                ] [elastic_src01_2] elastic/N8r-gD9WQSSvAYMOlJzmIg
[2016-02-28 12:38:39,796][INFO ][cluster.service          ] [elastic_src01_2] detected_master {elastic_src01_1}{UWGAo0BKTQm2f650nyDKYg}{172.21.0.21}{1
72.21.0.21:9300}, added {{elastic_src01_1}{UWGAo0BKTQm2f650nyDKYg}{172.21.0.21}{172.21.0.21:9300},{elastic_src01_1}{qNDQjkmsRjiIVjZ88JsX4g}{172.21.0.2
1}{172.21.0.21:9301},}, reason: zen-disco-receive(from master [{elastic_src01_1}{UWGAo0BKTQm2f650nyDKYg}{172.21.0.21}{172.21.0.21:9300}])
[2016-02-28 12:38:39,843][INFO ][http                     ] [elastic_src01_2] publish_address {172.21.0.21:9202}, bound_addresses {172.21.0.21:9202}
[2016-02-28 12:38:39,843][INFO ][node                     ] [elastic_src01_2] started

However, when I started Node 1 on SRC02 machine, I am not seeing detected_master message. Here is what ES generates:

[2016-02-28 12:22:52,256][INFO ][node                     ] [elastic_src02_1] version[2.2.0], pid[6432], build[8ff36d1/2016-01-27T13:32:39Z]
[2016-02-28 12:22:52,256][INFO ][node                     ] [elastic_src02_1] initializing ...
[2016-02-28 12:22:52,662][INFO ][plugins                  ] [elastic_src02_1] modules [lang-expression, lang-groovy], plugins [], sites []
[2016-02-28 12:22:52,693][INFO ][env                      ] [elastic_src02_1] using [1] data paths, mounts [[Data (E:)]], net usable_space [241.6gb], net total_
space [249.8gb], spins? [unknown], types [NTFS]
[2016-02-28 12:22:52,693][INFO ][env                      ] [elastic_src02_1] heap size [910.5mb], compressed ordinary object pointers [true]
[2016-02-28 12:22:54,193][INFO ][node                     ] [elastic_src02_1] initialized
[2016-02-28 12:22:54,193][INFO ][node                     ] [elastic_src02_1] starting ...
[2016-02-28 12:22:54,334][INFO ][transport                ] [elastic_src02_1] publish_address {172.21.0.22:9300}, bound_addresses {172.21.0.22:9300}
[2016-02-28 12:22:54,334][INFO ][discovery                ] [elastic_src02_1] elastic/SNvuAfnxQV-RW430zLF6Vg
[2016-02-28 12:22:58,912][INFO ][cluster.service          ] [elastic_src02_1] new_master {elastic_src02_1}{SNvuAfnxQV-RW430zLF6Vg}{172.21.0.22}{172.21.0.22:9300
}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-02-28 12:22:58,943][INFO ][gateway                  ] [elastic_src02_1] recovered [0] indices into cluster_state
[2016-02-28 12:22:58,959][INFO ][http                     ] [elastic_src02_1] publish_address {172.21.0.22:9200}, bound_addresses {172.21.0.22:9200}
[2016-02-28 12:22:58,959][INFO ][node                     ] [elastic_src02_1] started

Does the node on the SRC02 machine really form a cluster with nodes on the SRC01 machine?

2. On the same machine (SRC01), if I add

discovery.zen.minimum_master_nodes: 3

to the elasticsearch.yml file of Node elastic_src01_1, elastic_src01_2, then when starting the second node elastic_src01_2 on machine SRC01, I am unable to see detected_master in ES-generated messages.

Does this mean elastic_src01_1 and elastic_src01_2 not form a cluster?

Thanks for help!

UPDATE 2

The SRC01 and SRC02 machines can see each other. Here are ping results from SRC02 to SRC01:

C:\Users\Administrator>ping 172.21.0.21

Pinging 172.21.0.21 with 32 bytes of data:
Reply from 172.21.0.21: bytes=32 time<1ms TTL=128
Reply from 172.21.0.21: bytes=32 time<1ms TTL=128
Reply from 172.21.0.21: bytes=32 time<1ms TTL=128
Reply from 172.21.0.21: bytes=32 time<1ms TTL=128

UPDATE 3

The problem is resolved. The reason that my setup was not working before is the server's firewall prevented port 9300/9200 for communication.

Answer 1

Basically you simply need to configure the network settings to make sure that all nodes can see each other on the network. Additionally, since you're running two nodes on the same machine and you still want high availability, you want to prevent a primary shard and its replica to land on the same physical machine.

Finally, since you'll have a total of four nodes in your cluster, you'll want to prevent split brain situations, so you need to set discovery.zen.minimum_master_nodes as well.

Node 1/2 on SRC01:

# cluster name
cluster.name: Name_of_your_cluster

# Give each node a different name (optional but good practice if you don't know Marvel characters)
node.name: SRC01_Node1/2

# The IP that this node will bind to and publish
network.host: 172.21.0.21

# The IP of the other nodes
discovery.zen.ping.unicast.hosts: ["172.21.0.22"]

# prevent split brain
discovery.zen.minimum_master_nodes: 3    

# to prevent primary/replica shards to be on the same physical host 
# see why at http://stackoverflow.com/questions/35677741/proper-value-of-es-heap-size-for-a-dedicated-machine-with-two-nodes-in-a-cluster
cluster.routing.allocation.same_shard.host: true

# prevent memory swapping
bootstrap.mlockall: true

Node 1/2 on SRC02:

# cluster name
cluster.name: Name_of_your_cluster

# Give each node a different name (optional but good practice if you don't know Marvel characters)
node.name: SRC02_Node1/2

# The IP that this node will bind to and publish
network.host: 172.21.0.22

# The IP of the other nodes
discovery.zen.ping.unicast.hosts: ["172.21.0.21"]

# prevent split brain
discovery.zen.minimum_master_nodes: 3    

# to prevent primary/replica shards to be on the same physical host 
# see why at http://stackoverflow.com/questions/35677741/proper-value-of-es-heap-size-for-a-dedicated-machine-with-two-nodes-in-a-cluster
cluster.routing.allocation.same_shard.host: true

# prevent memory swapping
bootstrap.mlockall: true