How to manually purge data from Graylog 2.1

Question

I have a Graylog 2.1 server that has been running for some time. I hadn't paid attention to my retention rate recently and came in this morning to find Graylog partially crashed because the disk was out of space. Nearly 100% of the disk space is currently being taken up by Elasticsearch Shards. The web interface for Graylog is not currently usable in the state it's in. I tried some of the standard Ubuntu tricks for freeing up disk space like apt-get autoremove and clean, but wasn't able to get enough to get the web interface functional.

The problem is all of the documentation I can currently find for changing the retention rate and cycling the shards, is via the web interface. The only config options no longer appear present in the Graylog config file.

Does anyone know of a manual, CLI, way of purging data from the Elasticsearch Shards in Graylog 2.1?

Answer 1

First aid: check which indices are present:

curl http://localhost:9200/_cat/indices

Then delete the oldest indices (you should not delete all)

curl -XDELETE http://localhost:9200/graylog_1
curl -XDELETE http://localhost:9200/graylog_2
curl -XDELETE http://localhost:9200/graylog_3

Fix: You can then reduce the parameter elasticsearch_max_number_of_indices in /etc/graylog/server/server.conf to a value that fits your disk.

Answer 2

If Elasticsearch is still starting, you can simply delete indices with the Delete Index API, which is, after using Graylog directly (System / Indices page in the web interface), the preferred way of getting rid of Elasticsearch indices.

If you're totally screwed (i. e. neither Graylog, nor Elasticsearch are starting), you can still delete the complete data from Elasticsearch's data path (see Directory Layout).