AWS elastic-search. FORBIDDEN/8/index write (api). Unable to write to index

Question

I am trying dump a list of docs to an AWS elastic-search instance. It was running fine. Then, all of sudden it started throwing this error:

{ _index: '<my index name>',
  _type: 'type',
  _id: 'record id',
  status: 403,
  error: 
   { type: 'cluster_block_exception',
     reason: 'blocked by: [FORBIDDEN/8/index write (api)];' } }

I checked in forums. Most of them says that it is a JVM memory issue. If it is going more than 92%, AWS will stop any writes to the cluster/index. However, when I checked the JVM memory, it shows less than 92%. I am missing something here?

Answer 1

This error is the Amazon ES service actively blocking writes to protect the cluster from reaching red or yellow status. It does this using index.blocks.write.

The two reasons being:

Low Memory

When the JVMMemoryPressure metric exceeds 92% for 30 minutes, Amazon ES triggers a protection mechanism and blocks all write operations to prevent the cluster from reaching red status. When the protection is on, write operations fail with a ClusterBlockException error, new indexes can't be created, and the IndexCreateBlockException error is thrown.

When the JVMMemoryPressure metric returns to 88% or lower for five minutes, the protection is disabled, and write operations to the cluster are unblocked.

Low Disk Space

Elasticsearch has a default "low watermark" of 85%, meaning that once disk usage exceeds 85%, Elasticsearch no longer allocates shards to that node. Elasticsearch also has a default "high watermark" of 90%, at which point it attempts to relocate shards to other nodes.

Answer 2

This error indicates that AWS ElasticSearch has placed a block on your domain based upon disk space. At 85%, ES will not allow you create any new indexes. At 90%, no new documents can be written.

Answer 3

ES could apply write block on index during rollovers, or Low disk space or memory.

In order to stop these errors you need to remove the write block on the index by setting index.blocks.write to false

curl -X PUT -H "Content-Type: application/json" \
'http://localhost:9200/{index_name}/_settings' \
  -d '{ "index": { "blocks": { "write": "false" } } }'

Answer 4

The accepted solution was not enough in my case, I had to remove index.blocks.read_only_allow_delete as well

PUT /my_index/_settings
{
  "index.blocks.read_only_allow_delete": null,
  "index.blocks.write": null
}

ES version 7.15