I am using Filebeat to ship log data from my local txt files into Elasticsearch, and
I want to add some fields from the message
line to the event - like timestamp and log level. For example here is one of my log lines:
2016-09-22 13:51:02,877 INFO 'start myservice service'
My question is: Can I do that by Filebeat -> Elasticsearch or must I go through Logstash?
You can use Filebeat -> Elasticsearch if you make use of the Ingest Node feature in Elasticsearch 5.0. Otherwise, yes, you need to use Logstash.
In both cases you would use a grok filter to parse the message
line into structured data. Also you'll want to use a date
to parse and normalize the date.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With