ElasticSearch entered "read only" mode, node cannot be altered

Question

Something happened during the night to my ES cluster (composed of 5 data nodes, 3 master nodes).

I have no idea what happened but all the indices and data were deleted and the cluster entered a "read only" mode, possibly hacked?

When trying to get Kibana running, I get the following:

Tried restarting Kibana - it restarted, nothing changed. Tried restarting Elastic - it restarted (all nodes), nothing changed.

I then had a look at the cluster settings and this is what I got:

{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "all"
        }
      },
      "blocks": {
        "read_only": "true"
      }
    }
  },
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "all"
        }
      }
    }
  }
}

I tried undoing the read only as follows:

PUT _cluster/settings
{
  "persistent": {
    "blocks.read_only": false
  }
}

No luck as you can see:

{
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
      }
    ],
    "type": "cluster_block_exception",
    "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
  },
  "status": 403
}

Any ideas?

UPDATE: Problem solved by Andrei Stefan, now for the more important part - why? What happened and why? I've lost all data and my cluster entered a read-only mode.

Answer 1

The correct command is:

PUT /_cluster/settings
{
  "persistent" : {
    "cluster.blocks.read_only" : false
  }
}