I need to monitor very different log files for errors, success status etc. And I need to grab corresponding metrics using Prometheus and show in Grafana + set some alerting on it. Prometheus + Grafana are OK I already use them a lot with different exporters like node_exporter or mysql_exporter etc. Also alerting in new Grafana 4.x works very well.
But I have quite a problem to find suitable exporter/ program which could analyze log files "on fly" and extract metrics from them.
So far I tried:
Does any one here has a really running solution for monitoring advanced metrics from log files using "some exporter" + Prometheus + Grafana? Or instead of exporter some program from which I could grab results using Prometheus push gateway. Thanks.
The log files are matches line by line using a regular expression, and the matched data is stored in different variables. Then, the data of these variables are exported via grok_exporter. Once you add the grok_exporter server to Prometheus as a target, Prometheus can monitor the desired log files via grok_exporter.
If you use Prometheus and Grafana for metrics storage and data visualization, Solr includes a Prometheus exporter to collect metrics and other data.
After login into the Grafana, click on the Explore tab. Select data source as Loki and enter the {app=”nginx”} in Log labels. Note that metadata of Nginx deployment contains the label app=nginx which is the same that filters out logs in Loki. Alerts enter into the Pending state as soon as its condition satisfies.
Take a look at Telegraf. It does support tailing logs using input plugins logparser and tail. To export metrics as prometheus endpoint use prometheus_client output plugin. You also may apply on the fly aggregations. I've found it simpler to configure for multiple log files than grok_exporter or mtail
Those are the 3 answers currently for getting log data into Prometheus.
You could also look into getting whatever is producing the logs to expose Prometheus metrics directly.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With