ElasticSearch ILM not deleting indices

Question

I have set a simple ILM policy on my fluentd.* indices to be deleted after (for testing - ) a short period of time.

ILM:

PUT _ilm/policy/fluentd
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_size": "1gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "4d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Index Template:

PUT _template/fluentd
{
  "order": 0,
  "index_patterns": [
    "fluentd.*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "fluentd"
      }
    }
  },
  "aliases": {
    "fluent": {}
  }
}

With these settings, I expected ES to delete indices older than 5-6 days, but there are still indices from 3 weeks ago in ES. Currently, it says there are 108 linked indices to this ILM policy.

What is it actually doing, it seems it's not doing anything at all... how to delete indices after x days?

I tried first to use the "index template" but it's useless, it does not apply settings to each index (maybe yes but only on creation????).

Then I put the ILM on the index by hand (another bug: you can't select all index and hit "add ILM policy" - you need to add the policy one by one) which required me to click about 600 times.

Now the problem was, I had "hot" phase defined but it didn't trigger (it's buggy?) - because the hot phase didn't trigger (i set to to "rollover after 1 day after index creation") - the delete phase didn't either. When I removed the hot phase and applied the ILM to index again with only delete - it worked! but adding and removing all this is buggy, I get Ooops, something went wrong errors here and there.

I don't understand why I have to remove the ILM and reapply it to each index when I change something in the ILM policy. It's 1000% inconvenient.

ES really needs to put some work into it, it's still too beta and I got a hell lot of status code 500, although I am using most recent version directly on Elastic Cloud.

Answer 1

With these settings, I expected ES to delete indices older than 5-6 days, but there are still indices from 3 weeks ago in ES. Currently, it says there are 108 linked indices to this ILM policy.

With your settings, the delete phase starts at 4 day from rollover. If you want to start the delete phase at 4 day from "index creation" you need to remove the rollover action from the hot phase:

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "4d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

I tried first to use the "index template" but it's useless, it does not apply settings to each index (maybe yes but only on creation????).

Yes, it works on index creation.

Then I put the ILM on the index by hand (another bug: you can't select all index and hit "add ILM policy" - you need to add the policy one by one) which required me to click about 600 times.

Kibana does not allow you to apply ILM policy to all index, but the elasticsearch API allows it! Simply open a kibana dev tools and run the follow request:

PUT fluentd.*/_settings
{
  "index": {
    "lifecycle": {
      "name": "fluentd"
    }
  }
}

Now the problem was, I had "hot" phase defined but it didn't trigger (it's buggy?) - because the hot phase didn't trigger (i set to to "rollover after 1 day after index creation") - the delete phase didn't either. When I removed the hot phase and applied the ILM to index again with only delete - it worked! but adding and removing all this is buggy, I get Ooops, something went wrong errors here and there.

If rollover phase was not triggered, the ILM could not progress.

I don't understand why I have to remove the ILM and reapply it to each index when I change something in the ILM policy. It's 1000% inconvenient.

Because the ILM definition are cached on each index. see the doc: https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html#ilm-phase-execution

Answer 2

A little bit late, but maybe it will help somebody.

Another reason can be like that was mentioned here:

ILM is not really intended to be used on 1m lifecycle. I Do not believe You will achieve your desired behavior. My understanding is that ILM is an opportunistic background task it is not preemptive so it is not going to execute on the exact time frame.

It's designed to work on the order of hours or days not minutes.

I have the same situation at my indices and I checked - indices are deleted, but later than I sat them up.