How to delete a specific shard of an ElasticSearch index

Question

I recently had a SNAFU cause my cluster to end up with split-brain (despite having many controls in place) resulting in shards that are basically busted. I've got all the nodes back in play properly, recognizing the right master, etc. but the cluster remains red and rightfully so; there are a few shards that have no home.

After using my RubberBand script, I was able to explore using VisualJSON to find shards like the following one, that have no node:

{
    "index": "logstash-2013.12.27",
    "node": null,
    "primary": false,
    "relocating_node": null,
    "shard": 4,
    "state": "UNASSIGNED"
},

I would like to delete them but I can't seem to find an API call to delete a shard, only deleting whole indices or using queries. Thanks in advance!

Answer 1

This command will take an orphaned shard and assign it to node efsKb4DzQ2iaIfKfu36vsA.

curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
  "commands": [
    {
      "allocate": {
        "index": "tweedle-2013.12.21",
        "shard": 3,
        "node": "efsKb4DzQ2iaIfKfu36vsA",
        "allow_primary": true
      }
    }
  ]
}'

Answer 2

You can't delete an unassigned shard because there is no shard to be deleted. An unassigned shard is not a corrupted shard, but a missing replica.

Your config probably tells ES (ElasticSearch) to create replicas and assign them on different nodes for high availability and/or fault tolerance. ES was not able to automatically create and assign a replica and, thus, you see the UNASSIGNED state. It could have been due to a network error, memory not available, etc.

You may want to find the reason why the allocation failed:

curl -XPOST 'localhost:9200/_cluster/allocation/explain?pretty'

And, then, ask ES to retry the allocation for you:

curl -XPOST 'localhost:9200/_cluster/reroute?retry_failed'

---

Credits to ES's expert answer which says

After 5 unsuccessful allocation attempts, the master gives up and needs manual triggering to give it another allocation attempt