Removing old indices in elasticsearch

Question

I have the many of my logs indexed in logstash-Year-Week format. That is if i want to delete indices older than a few weeks, how can I achieve that in elasticsearch. Is there an easy, seamless way to do that?

Answer 1

Curator would be an ideal match here. You can find the link here - https://github.com/elastic/curator

A command like below should work just fine -

curator --host <IP> delete indices --older-than 30 --prefix "twitter-" --time-unit days  --timestring '%Y-%m-%d' 

You can keep in this in the CRON for removing the indices occasionally.

You can find some examples and docs here - https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html

Answer 2

If you are using elasticsearch version 5.x then you need to install the curator version 4.x. You can see the version compatibility and installation steps from the documentation

Once installed. Then just run the command

curator --config path/config_file.yml [--dry-run] path/action_file.yml 

Curator provides a dry-run flag to just output what Curator would have executed. Output will be in your log file which you have defined in config.yml file. If not logging key defined in config_file.yml then currator will output to console. To delete the indices run the above command without --dry-run flag

The configuration file config_file.yml is

--- client:   hosts:    - 127.0.0.1   port: 9200 logging:   loglevel: INFO   logfile: "/root/curator/logs/actions.log"   logformat: default   blacklist: ['elasticsearch', 'urllib3'] 

The action file action_file.yml is

--- actions:   1:     action: delete_indices     description: >-       Delete indices older than 7 days (based on index name), for logstash-       prefixed indices. Ignore the error if the filter does not result in an       actionable list of indices (ignore_empty_list) and exit cleanly.     options:       ignore_empty_list: True       timeout_override:       continue_if_exception: False       disable_action: False     filters:     - filtertype: pattern       kind: prefix       value: logstash-       exclude:     - filtertype: age       source: name       direction: older       timestring: '%Y.%m.%d'       unit: days       unit_count: 7       exclude: 

If you want to delete the indices weekly, monthly, etc automatically. Then just write the bash script like

#!/bin/bash # Script to delete the log event indices of the elasticsearch weekly  #This will delete the indices of the last 7 days curator --config /path/config_file.yml /path/action_file.yml 

Put a shell script in one of these folders: /etc/cron.daily, /etc/cron.hourly, /etc/cron.monthly or /etc/cron.weekly and your job is done.

NOTE: Make sure to use the correct indentation in your configuration and action files. Otherwise it will not work.