AWS ElasticSearch: How to apply a policy to an index

Question

We have an AWS ElasticSearch domain and are writing records/documents to it. I've now created an Index State/Lifecycle Management (ISM/ILM) policy in Kibana and I can apply the policy to an Index from within Kibana. I now want to apply that policy when the index is created from within our Java code that handles writes to the Index (using the High Level REST API).

I have found no methods in the High Level REST API that specifically allow assigning a policy to an index, however I think that I should be able to do it using the RequestOptions object that is used when the Index is created. The documentation is pretty thin, but it seems that I should be able to basically insert a key/value into the Index properties. For example, when I inspect the index where I have manually assigned the policy, find the following keys where the policy is assigned.

"settings" : {
  "index" : {
    "opendistro" : {
      "index_state_management" : {
        "policy_id" : "DefaultLifeCyclePolicy_30DayWarm_180DayDelete"
      }
    },

It seems reasonable to assume that i can just insert a similar key into the Index object. The following code seems like it should work. It does run without error, but the RequestOptions does nothing useful.

        boolean isExisting = mAwsClient.indices().exists(new GetIndexRequest(indexNameFull), RequestOptions.DEFAULT);
        if (!isExisting) {
            RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
            builder.addHeader("settings.opendistro.index_state_management.policy_id", mIndexStateMgmtPolicy);
            RequestOptions requestOptions = builder.build();

            CreateIndexRequest request = new CreateIndexRequest(indexNameFull);
            request.mapping(mapping, XContentType.JSON);
            CreateIndexResponse createIndexResponse = mAwsClient.indices().create(request, requestOptions);
        }

So, how can I assign am ISM/ILM policy to an index using Java?

BTW, I have looked at creating an Index Template, which should assign the policy automatically on index creation, but the OpenDistro Kibana does not seem to have that functionality.

Answer 1

BTW, I have looked at creating an Index Template, which should assign the policy automatically on index creation, but the OpenDistro Kibana does not seem to have that functionality.

You can use the index template in the following manner to apply ISM policy at index creation:

PUT _template/template_1
{
  "index_patterns": [
    "test-index*"
  ],
  "settings": {
    "index": {
      "opendistro": {
        "index_state_management": {
          "policy_id": "DefaultLifeCyclePolicy_30DayWarm_180DayDelete"
        }
      }
    }
  }
}

For rollover indices:

1. Make sure you have alias:

POST /_aliases
{
    "actions" : [
        { "add" : { "index" : "test-index-000001", "alias" : "test-index" } }
    ]
} 

2. Template:

PUT _template/template_1
{
  "index_patterns": [
    "test-index*"
  ],
  "settings": {
    "index": {
      "opendistro": {
        "index_state_management": {
          "policy_id": "DefaultLifeCyclePolicy_30DayWarm_180DayDelete",
          "rollover_alias": "test-index"
        }
      }
    }
  }
}

Answer 2

The above answer is correct but the policy_id in template is deprecated.In new version this is not applicable.

In new version you have to add index pattern in the policy itself as below example.

PUT _opendistro/_ism/policies/policy_name
{
  "policy": {
    "description": "Policy to manage indices",
    "default_state": "hot",
    "states" : [
      {
        "name" : "hot",
        "actions" : [
          {
            "rollover" : {
              "min_size" : "20gb",
              "min_index_age" : "2d"
            }
          }
        ]
      }
    ],
    "ism_template": {
      "index_patterns": [
        "nginx-error-logs*",   // **sample index pattern**
        "nginx-access-logs*"
      ],
      "priority": 100
    }
  }
}

Whenever new index create, the index name pattern will match to the ism_template and the respective policy will be applied.

If same pattern available in multiple policy the it will attach the policy who has high prority.