Elasticsearch read_only_allow_delete auto setting

Question

I have problem with Elasticsearch. I tried the following:

 $ curl -XPUT -H "Content-Type: application/json" \
     http://localhost:9200/_all/_settings \
       -d '{"index.blocks.read_only_allow_delete": false}'

My settings:

"settings": {
  "index": {
    "number_of_shards": "5",
    "blocks": {
      "read_only_allow_delete": "true"
    },
    "provided_name": "new-index",
    "creation_date": "1515433832692",
    "analysis": {
      "filter": {
        "ngram_filter": {
          "type": "ngram",
          "min_gram": "2",
          "max_gram": "4"
        }
      },
      "analyzer": {
        "ngram_analyzer": {
          "filter": [
            "ngram_filter"
          ],
          "type": "custom",
          "tokenizer": "standard"
        }
      }
    },
    "number_of_replicas": "1",
    "uuid": "OSG7CNAWR9-G3QC75K4oQQ",
    "version": {
      "created": "6010199"
    }
  }
}

When I check settings it looks fine, but only a few seconds (3-5) and it's still set to true. I can't add new elements and query anything, only _search and delete.

Someone have any idea how to resolve this?

NOTE: I'm using Elasticsearch version: 6.1.1

Answer 1

Elasticsearch automatically sets "read_only_allow_delete": "true" when hard disk space is low.

Find the files which are filling up your storage and delete/move them. Once you have sufficient storage available run the following command through the Dev Tool in Kibana:

PUT your_index_name/_settings
{
 "index": {
   "blocks": {
     "read_only_allow_delete": "false"
    }
  }
}

OR (through the terminal):

$ curl -XPUT -H "Content-Type: application/json" \
   http://localhost:9200/_all/_settings \
     -d '{"index.blocks.read_only_allow_delete": false}'

as mentioned in your question.

Answer 2

In an attempt to add a sprinkling of value to the accepted answer (and because i'll google this and come back in future), for my case the read_only_allow_delete flag was set because of the default settings for disk watermark being percentage based - which on my large disk did not make as much sense. So I changed these settings to be "size remaining" based as the documentation explains.

So before setting read_only_allow_delete back to false, I first set the watermark values based on disk space:

(using Kibana UI):

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "20gb",
    "cluster.routing.allocation.disk.watermark.high": "15gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "10gb"
  }
}

PUT your_index_name/_settings
{
 "index": {
   "blocks": {
     "read_only_allow_delete": "false"
    }
  }
}

OR (through the terminal):

$ curl -XPUT -H "Content-Type: application/json" \
   http://localhost:9200/_cluster/_settings \
   -d '{"cluster.routing.allocation.disk.watermark.low": "20gb", 
     "cluster.routing.allocation.disk.watermark.high": "15gb", 
     "cluster.routing.allocation.disk.watermark.flood_stage": "10gb"}'

$ curl -XPUT -H "Content-Type: application/json" \
   http://localhost:9200/_all/_settings \
   -d '{"index.blocks.read_only_allow_delete": false}'