Changing elasticsearch index's shard-count on the next index-rotation

Question

I have an ELK (Elasticsearch-Kibana) stack wherein the elasticsearch node has the default shard value of 5. Logs are pushed to it in logstash format (logstash-YYYY.MM.DD), which - correct me if I am wrong - are indexed date-wise.

Since I cannot change the shard count of an existing index without reindexing, I want to increase the number of shards to 8 when the next index is created. I figured that the ES-API allows on-the-fly persistent changes.

How do I go about doing this?

Answer 1

You can use the "Template Management" features in Elasticsearch: http://www.elasticsearch.org/guide/en/elasticsearch/reference/1.x/indices-templates.html

Create a new logstash template by using:

curl -XPUT localhost:9200/_template/logstash -d '
{
  "template": "logstash-*",
  "settings": {
    "number_of_replicas": 1,
    "number_of_shards": 8,
    "index.refresh_interval": "5s"
  },
  "mappings": {
    "_default_": {
      "_all": {
        "enabled": true
      },
      "dynamic_templates": [
        {
          "string_fields": {
            "match": "*",
            "match_mapping_type": "string",
            "mapping": {
              "type": "string",
              "index": "analyzed",
              "omit_norms": true,
              "fields": {
                "raw": {
                  "type": "string",
                  "index": "not_analyzed",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      ],
      "properties": {
        "@version": {
          "type": "string",
          "index": "not_analyzed"
        },
        "geoip": {
          "type": "object",
          "dynamic": true,
          "path": "full",
          "properties": {
            "location": {
              "type": "geo_point"
            }
          }
        }
      }
    }
  }
}'

The next time the index that matches your pattern is created, it will be created with your new settings.

Answer 2

The setting is on your elasticsearch. You need to change to config file config/elasticsearch.yml

Change the index.number_of_shards: 8. and restart elasticsearch. The new configuration will set and the new index will use the new configuration, which create 8 shard as you want.