Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can I delete the message field from Logstash?

Tags:

I have a basic Logstash -> Elasticsearch setup, and it turns out the 'message' field is not required after the logstash filter done its job - storing this raw message field to elasticsearch is only adding unnecessary data to storage imo.

Can I safely delete this field and would it cause any trouble to ES? advices or readings are welcome, thanks all.

like image 535
James Jiang Avatar asked Sep 24 '14 00:09

James Jiang


People also ask

How do I delete a field in Logstash?

You can remove these using the mutate filter plugin. There are several different ways of using this plugin to cover a wide range of use-cases, and so it is important to choose the right strategy depending on your situation.

How do I delete a field in Kibana?

You have to delete and re-index the indices still containing the field with the updated mapping. Then you can go to the index pattern management page in Kibana and refresh the index pattern there using the reload button in the top right. This should remove the fields from the index pattern as well.

How does Logstash communicate with Elasticsearch?

Logstash receives these events by using the Beats input plugin for Logstash and then sends the transaction to Elasticsearch by using the Elasticsearch output plugin for Logstash. The Elasticsearch output plugin uses the bulk API, making indexing very efficient.

What can Logstash do?

Logstash allows you to easily ingest unstructured data from a variety of data sources including system logs, website logs, and application server logs.


2 Answers

No, it will not cause any trouble to ES. You can delete message field if it is redundant or unused.

You can add this filter to end of the filters.

mutate {      remove_field => [ "message" ] } 
like image 143
Ben Lim Avatar answered Oct 02 '22 09:10

Ben Lim


You can also do this within the json filter.

filter {   json {     source => "message"     remove_field => ["message"]   } } 
like image 42
Steve Avatar answered Oct 02 '22 07:10

Steve