Graylog2- how to config logs retention to 1 week

Question

We are using some Graylog2 servers ( graylog-server version 1.3.4). Because we receive too much of log messages, it requires a lot of memory. I am trying to reduce the logs retention to 1 week, every log messages older than 1 week will be deleted. However, I cannot find out any value in configuration file to do that.
I used "max_time_per_index = 7d" value but max_time_per_index seems just define the age of an index until it's rotated and a new index is being created, not of the messages in that index.
So, what's the best way to set message retention to 1 week? Please help me. Thanks a lot.

Answer 1

This can be easily configured using the Web GUI in Graylog_2 and later.

Navigate to "System/Indices" in the Administration drop down menu. Under "Settings", click the Update configuration button.

Configure the Index Rotation Configuration to equal "Index Time", Rotation Period = P1D (a day). You'll have to decide whether or not you'd like to "Delete Index" or just close it, then set the Max number of indices to "8". That should keep the current day, and the last 7 days worth of indices.

NOTE:

Graylog Enterprise edition comes with an option to "Archive" log files, which essentially compresses them and allows you to move it to another storage location (whether to tape or just to another storage location).

Answer 2

One way to achieve this is to rotate your index every day and keep the maximum number of index to 8. This way you will always have a full week + the current day of logs in your Elasticsearch cluster.

elasticsearch_max_time_per_index = 1d
elasticsearch_max_number_of_indices = 8

Note that your search performance can be even better by using more index and less rotation time thanks to the intelligent time range selection feature of Graylog. For example, this should give faster search results if you have many datas :

elasticsearch_max_time_per_index = 12h
elasticsearch_max_number_of_indices = 16

you can even drop your number of indices to 15 and still have a whole week of data.