Elasticsearch doesn't allow to allocate unassigned shard

Question

I have an ES cluster of 2 nodes. As I restarted nodes the cluster status is yellow as some of the shards are unassigned. I've tried to google and the common solution is to reroute unassigned shards. Unfortunately, it doesn't work for me.

curl localhost:9200/_cluster/health?pretty=true
{
  "cluster_name" : "infra",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 34,
  "active_shards" : 68,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 31,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 68.68686868686868
}

curl localhost:9200/_cluster/settings?pretty
{
  "persistent" : { },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all"
        }
      }
    }
  }
}

curl localhost:9200/_cat/indices?v

health status index                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash-log-2016.05.13   5   2      88314            0    300.5mb        150.2mb
yellow open   logstash-log-2016.05.12   5   2     254450            0    833.9mb        416.9mb
yellow open   .kibana                   1   2          3            0     47.8kb         25.2kb
green  open   .marvel-es-data-1         1   1          3            0      8.7kb          4.3kb
yellow open   logstash-log-2016.05.11   5   2     313095            0    709.1mb        354.6mb
yellow open   logstash-log-2016.05.10   5   2     613744            0        1gb        520.2mb
green  open   .marvel-es-1-2016.05.18   1   1      88720          495     89.9mb           45mb
green  open   .marvel-es-1-2016.05.17   1   1      69430          492     59.4mb         29.7mb
yellow open   logstash-log-2016.05.17   5   2     188924            0    518.2mb          259mb
yellow open   logstash-log-2016.05.18   5   2     226775            0    683.7mb        366.1mb

Rerouting

curl -XPOST 'localhost:9200/_cluster/reroute?pretty' -d '{
     "commands": [
        {
            "allocate": {
                "index": "logstash-log-2016.05.13",
                "shard": 3,
                "node": "elasticsearch-mon-1",
                "allow_primary": true
          }
        }
    ]
  }'
{
  "error" : {
    "root_cause" : [ {
      "type" : "illegal_argument_exception",
      "reason" : "[allocate] allocation of [logstash-log-2016.05.13][3] on node {elasticsearch-mon-1}{K-J8WKyZRB6bE4031kHkKA}{172.45.0.56}{172.45.0.56:9300} is not allowed, reason: [YES(allocation disabling is ignored)][NO(shard cannot be allocated on same node [K-J8WKyZRB6bE4031kHkKA] it already exists on)][YES(no allocation awareness enabled)][YES(allocation disabling is ignored)][YES(target node version [2.3.2] is same or newer than source node version [2.3.2])][YES(primary is already active)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(shard not primary or relocation disabled)][YES(node passes include/exclude/require filters)][YES(enough disk for shard on node, free: [25.4gb])][YES(below shard recovery limit of [2])]"
    } ],
    "type" : "illegal_argument_exception",
    "reason" : "[allocate] allocation of [logstash-log-2016.05.13][3] on node {elasticsearch-mon-1}{K-J8WKyZRB6bE4031kHkKA}{172.45.0.56}{172.45.0.56:9300} is not allowed, reason: [YES(allocation disabling is ignored)][NO(shard cannot be allocated on same node [K-J8WKyZRB6bE4031kHkKA] it already exists on)][YES(no allocation awareness enabled)][YES(allocation disabling is ignored)][YES(target node version [2.3.2] is same or newer than source node version [2.3.2])][YES(primary is already active)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(shard not primary or relocation disabled)][YES(node passes include/exclude/require filters)][YES(enough disk for shard on node, free: [25.4gb])][YES(below shard recovery limit of [2])]"
  },
  "status" : 400
}

curl -XPOST 'localhost:9200/_cluster/reroute?pretty' -d '{
     "commands": [
        {
            "allocate": {
                "index": "logstash-log-2016.05.13",
                "shard": 3,
                "node": "elasticsearch-mon-2",
                "allow_primary": true
          }
        }
    ]
  }'
{
  "error" : {
    "root_cause" : [ {
      "type" : "illegal_argument_exception",
      "reason" : "[allocate] allocation of [logstash-log-2016.05.13][3] on node {elasticsearch-mon-2}{Rxgq2aWPSVC0pvUW2vBgHA}{172.45.0.166}{172.45.0.166:9300} is not allowed, reason: [YES(allocation disabling is ignored)][NO(shard cannot be allocated on same node [Rxgq2aWPSVC0pvUW2vBgHA] it already exists on)][YES(no allocation awareness enabled)][YES(allocation disabling is ignored)][YES(target node version [2.3.2] is same or newer than source node version [2.3.2])][YES(primary is already active)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(shard not primary or relocation disabled)][YES(node passes include/exclude/require filters)][YES(enough disk for shard on node, free: [25.4gb])][YES(below shard recovery limit of [2])]"
    } ],
    "type" : "illegal_argument_exception",
    "reason" : "[allocate] allocation of [logstash-log-2016.05.13][3] on node {elasticsearch-mon-2}{Rxgq2aWPSVC0pvUW2vBgHA}{172.45.0.166}{172.45.0.166:9300} is not allowed, reason: [YES(allocation disabling is ignored)][NO(shard cannot be allocated on same node [Rxgq2aWPSVC0pvUW2vBgHA] it already exists on)][YES(no allocation awareness enabled)][YES(allocation disabling is ignored)][YES(target node version [2.3.2] is same or newer than source node version [2.3.2])][YES(primary is already active)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][YES(shard not primary or relocation disabled)][YES(node passes include/exclude/require filters)][YES(enough disk for shard on node, free: [25.4gb])][YES(below shard recovery limit of [2])]"
  },
  "status" : 400
}

So it fails and doesn't make any change. Shards are still in unassigned state.

Thank you.

Added

curl localhost:9200/_cat/shards

logstash-log-2016.05.13 2 p STARTED     17706  31.6mb 172.45.0.166 elasticsearch-mon-2
logstash-log-2016.05.13 2 r STARTED     17706  31.5mb 172.45.0.56  elasticsearch-mon-1
logstash-log-2016.05.13 2 r UNASSIGNED
logstash-log-2016.05.13 4 p STARTED     17698  31.6mb 172.45.0.166 elasticsearch-mon-2
logstash-log-2016.05.13 4 r STARTED     17698  31.4mb 172.45.0.56  elasticsearch-mon-1
logstash-log-2016.05.13 4 r UNASSIGNED

Answer 1

For all the indices that are yellow you have configured 2 replicas:

health status index                   pri rep
yellow open   logstash-log-2016.05.13   5   2
yellow open   logstash-log-2016.05.12   5   2
yellow open   .kibana                   1   2
yellow open   logstash-log-2016.05.11   5   2
yellow open   logstash-log-2016.05.10   5   2
yellow open   logstash-log-2016.05.17   5   2
yellow open   logstash-log-2016.05.18   5   2

2 replicas on two nodes cluster is impossible. You need a third node for all the replicas to be assigned.

Or, decrease the number of replicas:

PUT /logstash-log-*,.kibana/_settings
{
  "index": {
    "number_of_replicas": 1
  }
}