Elasticsearch Docker: flood stage disk watermark [95%] exceeded

Question

I have a problem with my Elasticsearch nodes running in a docker environment. I'm starting them up with docker-compose and after a few minutes they tell me: flood stage disk watermark [95%] exceeded

I'm running it on a cluster with rather high storage capacity and I already tried to increase the watermark settings in the elasticsearch.yml file, but I still get the error. Maybe it has to do with the size of the docker containers.

Does anyone know what could be the problem? Any help is much appreciated.

The docker-compose.yml for reference:

version: '3.4'
services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: es01
    environment:
      #- discovery.type=single-node
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.1
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:7.8.1
    container_name: kib01
    depends_on:
      - es01
      - es02
      - es03
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: http://es01:9200
    networks:
      - elastic

  client:
    image: appropriate/curl:latest
    depends_on:
      - es01
      - es02
      - es03
    networks:
      - elastic
    command: sh -c "curl es01:9200 && curl kib01:5601"

  dash_app:
    build: .
    ports:
    - 0.0.0.0:8050:8050
    depends_on:
      - es01
      - es02
      - es03
      - kib01
    networks:
      - elastic

#mapping:
#  image: appropriate/curl:latest
#  depends_on:
#    - es01
#    - es02
#    - es03
#  networks:
#    - elastic
#  command:    "curl -v -XPUT 'es01:9200/urteile' -H 'Content-Type: application/json' -d '
#       {
#         'mappings': {
#           'properties': {
#             'date': {
#               'type': 'date'
#             }
#           }
#         }
#       }
#    '"

  #web:
   # build: .
   # ports:
    #  - 8000:8000
    #depends_on:
    #  - es01
    #  - es02
    #  - es03
    #networks:
    #  - elastic


volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: bridge

And docker info:

Server:
 Containers: 6
  Running: 3
  Paused: 0
  Stopped: 3
 Images: 185
 Server Version: 19.03.12
 Storage Driver: overlay
  Backing Filesystem: extfs
  Supports d_type: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc nvidia
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.7.2-kd-cluster
 Operating System: Debian GNU/Linux 9 (stretch)
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 125.8GiB
 Name: dpl01
 ID: KBGO:2E6L:NIHR:UQAL:K5CN:XWBI:R7TK:WWZF:MZBT:BCHE:HUQW:UKKM
 Docker Root Dir: /data/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Answer 1

I found the solution. The problem has to do with the disk usage in total as described in the answer from sastorsl here: low disk watermark [??%] exceeded on

The storage of the cluster I worked on was 98% used. While there were 400GB free, Elasticsearch only looks at the percentages, thus shutting down any write permissions of indices.

The solution is to manually set the watermarks after the nodes have started (setting them in the elasticsearch.yml didn't work for me for some reason):

curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_cluster/settings -d '{ "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } }'
curl -XPUT -H "Content-Type: application/json" http://localhost:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Of course you have to put in your index names. After that, the indices will be writable again.