Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Delete documents older than 30 days in elasticsearch [closed]

I want to delete documents in my elasticsearch index which are older than 30 days.

Any ideas?

EDIT:

I want this to happen automatically - no document in my index shoudl be older than 30 days. So, in my opinion there are 2 options: either using curator or DELETE requests.

I have tried both, but i failed. Somehow i have to create a filter which filters all documents older than 30 days and deletes them, when i am using DELETE http statement.

I tried with curator, but curator (as far as i understood this) deletes only whole indices. When attempting to delete indices older than 30 days with curator, my timestamp causing errors.My moment.js pattern looks like this"MMMM Do YYYY, HH:mm:ss.SSS".

EDIT 2: I added the following to my logstash configuration:

elasticsearch
    {
    hosts => ["http://localhost:9200"]
    index => "logstash-%{type}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
    }

So logstash creates for every type and every day a particular index. Now i can use curator to delete the indices older than a specific date.

Problem solved imho.

like image 869
ACKflow Avatar asked Mar 29 '16 15:03

ACKflow


People also ask

How do I delete a document from Elasticsearch?

You use DELETE to remove a document from an index. You must specify the index name and document ID. You cannot send deletion requests directly to a data stream. To delete a document in a data stream, you must target the backing index containing the document.

Does deleting index delete documents Elasticsearch?

Deleting an index deletes its documents, shards, and metadata. It does not delete related Kibana components, such as data views, visualizations, or dashboards. You cannot delete the current write index of a data stream.


Video Answer


1 Answers

You can use DELETE query for that: https://www.elastic.co/guide/en/elasticsearch/reference/1.6/docs-delete-by-query.html in example the query will delete everything older than: 2016-02-29

DELETE index_name/_query
{
  "query": {
    "filtered": {
      "query": {
        "query_string": {
          "query": "*"
        }
      },
      "filter": {
        "range": {
          "@timestamp": {
            "lte": "2016-02-29"
          }
        }
      }
    }
  }
}

Update >6.4

According to the official documentation, this function has been deprecated and replaced by _delete_by_query

POST index_name/_delete_by_query
{
  "query": { 
    "match": {
      "message": "some message"
    }
  }
}

https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html

like image 104
Кирилл Полищук Avatar answered Nov 15 '22 20:11

Кирилл Полищук