Can I use Kibana to parse the message field

Question

We are using ELK and shoving all syslogs into Elasticsearch.

I have a log type like whose message field looks like:

"message":"11/04/2016 12:04:09 PM|There are now 8 active connections#015"

I would like to use Kibana to parse the message to get the number of active connections over time and then graph that in Kibana.

Am I thinking of how to do this correctly? The reading I've done seems to be telling me to set up a filter in Logstash...but that seems like the wrong place to parse the message field for this single log line type, given the amount of messages/logs and message/log types getting sent through Logstash.

Is there a way to parse the message field for this number and then graph that count over time in Kibana?

Answer 1

Kibana is not meant to do this kind of parsing. There are a few options you can use:

1. You could write an analyser that analyses this string. It can be done, but I would not do it like this.
2. Use logstash, but you already suggested that yourself. If you feel log stash is to heavy and you have a choice for the version to use, go for option three.
3. Use ingest, this is a new feature of elasticsearch. This is kind of a lightweight logstash that comes pre-packaged with elastic, it support patterns with grok that can do this.