How to delete data from a specific index in elasticsearch after a certain period?

Question

I have an index in elasticsearch with is occupied by some json files with respected to timestamp. I want to delete data from that index.

curl -XDELETE http://localhost:9200/index_name

Above code deletes the whole index. My requirement is to delete certain data after a time period(for example after 1 week). Could I automate the deletion process?

I tried to delete by using curator.

But I think it deletes the indexes created by timestamp, not data with in an index. Can we use curator for delete data within an index?

It will be pleasure if I get to know that either of following would work:

• Can Curl Automate to delete data from an index after a period?
• Can curator Automate to delete data from an index after a period?
• Is there any other way like python scripting to do the job?

References are taken from the official site of elasticsearch.

Thanks a lot in advance.

Answer 1

You can use the DELETE BY QUERY API: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html

Basically it will delete all the documents matching the provided query:

POST twitter/_delete_by_query
{
  "query": { 
    "match": {
      "message": "some message"
    }
  }
}

But the suggested way is to implement indexes for different periods (days for example) and use curator to drop them periodically, based on the age:

...
logs_2019.03.11
logs_2019.03.12
logs_2019.03.13
logs_2019.03.14

Answer 2

Simple example using Delete By Query API:

POST index_name/_delete_by_query
{
  "query": {
    "bool": {
      "filter": {
        "range": {
          "timestamp": {
            "lte": "2019-06-01 00:00:00.0",
            "format": "yyyy-MM-dd HH:mm:ss.S"
          }
        }
      }
    }
  }
}

This will delete records which have a field "timestamp" which is the date/time (within the record) at which they occured. One can run the query to get a count for what will be deleted.

GET index_name/_search
{
  "size": 1,
  "query: {
-- as above --

Also it is nice to use offset dates

         "lte": "now-30d",

which would delete all records older than 30 days.