Why does Elasticsearch have a terrible performance indexing small amount of data inside docker?

Question

I try to configure django app with elasticsearch inside docker using docker-compose. Building of a small index takes around 15 minutes inside docker. The same command executes in 30 seconds if I run it outside docker.

Here is my docker-compose.yml which was based on the official docker installation guide:

version: '3'


services:

  web:
    build:
      context: ../..
      dockerfile: compose/local/Dockerfile
    restart: on-failure
    volumes:
      - ../..:/var/www/chesno
    env_file:
      - ../../.env.local
    depends_on:
      - elasticsearch1
    networks:
      - esnet
      - nginx_net

  nginx:
    image: "nginx:1.17.6-alpine"
    restart: always
    volumes:
      - ./nginx/conf.d:/etc/nginx/conf.d
    ports:
      - "5000:80"
    depends_on:
      - web
    networks:
      - nginx_net


  elasticsearch1:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.5.3
    container_name: elasticsearch
    environment:
      - node.name=chesno-node
      - cluster.name=chesno-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata:/usr/share/elasticsearch/data
    ports:
      - 9201:9200
      - 9301:9300
    networks:
      - esnet


volumes:
  esdata:
    driver: local


networks:
  esnet:
    driver: bridge
  nginx_net:
    driver: bridge

Command docker-compose docker-compose.yml exec elasticsearch1 curl -XGET http://localhost:9200/_cluster/health?pretty=true returns:

{
  "cluster_name" : "chesno-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 22,
  "active_shards" : 22,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 22,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}

The command utilizes only a fraction of machine's CPU and memory inside docker. Also, It has more shards comparing to the default elasticsearch setup for the project outside docker (that has only 5 shards).

Answer 1

I don't remember how I resolved this problem almost a year ago, but I have some ideas that might help. There are several problems with that setup:

1. Official instructions describe a multi-node cluster that consists of three nodes. For single-node cluster you should specify discovery.type=single-node. Single node cluster is suitable only for a development environment. For production, I suggest to leave docker and set up a multiple-server cluster with ansilbe.
2. It's better to use a recent version of elasticsearch
3. There are too many shards

A good practice is to ensure the amount of shards for each node stays below 20 per GB of heap that is configured.

Check out this tutorial to find out more.

4. Make sure that you have enough space on hard disk and don't receive an error flood stage disk watermark [95%] exceeded on.

Here is my current setup for elasticsearch:

services:
 
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    container_name: es01
    environment:
      - node.name=es01_local
      - cluster.name=es_cluster_local
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet

Command docker-compose docker-compose.yml exec es01 curl -XGET http://localhost:9200/_cluster/health?pretty=true returns:

{
  "cluster_name" : "es_cluster_local",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 6,
  "active_shards" : 6,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 6,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}