Menu
I have a basic Logstash -> Elasticsearch setup, and it turns out the 'message' field is not required after the logstash filter done its job - storing this raw message field to elasticsearch is only adding unnecessary data to storage imo.
Can I safely delete this field and would it cause any trouble to ES? advices or readings are welcome, thanks all.
535
You can remove these using the mutate filter plugin. There are several different ways of using this plugin to cover a wide range of use-cases, and so it is important to choose the right strategy depending on your situation.
You have to delete and re-index the indices still containing the field with the updated mapping. Then you can go to the index pattern management page in Kibana and refresh the index pattern there using the reload button in the top right. This should remove the fields from the index pattern as well.
Logstash receives these events by using the Beats input plugin for Logstash and then sends the transaction to Elasticsearch by using the Elasticsearch output plugin for Logstash. The Elasticsearch output plugin uses the bulk API, making indexing very efficient.
Logstash allows you to easily ingest unstructured data from a variety of data sources including system logs, website logs, and application server logs.
No, it will not cause any trouble to ES. You can delete message field if it is redundant or unused.
You can add this filter to end of the filters.
mutate { remove_field => [ "message" ] } You can also do this within the json filter.
filter { json { source => "message" remove_field => ["message"] } } If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
