Creating separate elasticsearch nodes of master+data and client+data

Question

We currently have a beefy 12 node cluster where all the nodes are master eligible, clients, and data. Recently we hit a split brain issue where the master was unresponsive long enough that another node elected itself. This despite all the nodes being up and minimum_master_nodes set to 7 (the new master could see 10 other eligible nodes). We'd like to reduce the risk of this by setting just 3 of the nodes to be master eligible and the other 9 to be clients, while all 12 remain data nodes. I had assumed we could just set:

node.master: true
node.client: false
node.data: true

on 3 of the nodes and:

node.master: false
node.client: true
node.data: true

on the other 9. That resulted in:

"org.elasticsearch.ElasticsearchIllegalStateException: node is not configured to store local location" 

I've been searching to no avail. Is there a right way to accomplish this?

Answer 1

I'd skip using node.client - it actually conflicts with your node.data setting and this is what is causing the error message. It's also not required.

You only need node.master and node.data. If both are true (the default) it's both a data node and a master node. If master is false and data is true it's a data only node. If master is true and data is false it's a master only node. And if both are false it's a client node.

From the current default elasticsearch.yml file for version 1.2.1:

# You can exploit these settings to design advanced cluster topologies.
#
# 1. You want this node to never become a master node, only to hold data.
#    This will be the "workhorse" of your cluster.
#
#node.master: false
#node.data: true
#
# 2. You want this node to only serve as a master: to not store any data and
#    to have free resources. This will be the "coordinator" of your cluster.
#
#node.master: true
#node.data: false
#
# 3. You want this node to be neither master nor data node, but
#    to act as a "search load balancer" (fetching data from nodes,
#    aggregating results, etc.)
#
#node.master: false
#node.data: false