Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Monitoring log files using some metrics exporter + Prometheus + Grafana

I need to monitor very different log files for errors, success status etc. And I need to grab corresponding metrics using Prometheus and show in Grafana + set some alerting on it. Prometheus + Grafana are OK I already use them a lot with different exporters like node_exporter or mysql_exporter etc. Also alerting in new Grafana 4.x works very well.

But I have quite a problem to find suitable exporter/ program which could analyze log files "on fly" and extract metrics from them.

So far I tried:

  • mtail (https://github.com/google/mtail) - works but existing version cannot easily monitor more files - in general it cannot bind specific mtail program (receipt for analysis) to some specific log file + I cannot easily add log file name into tag
  • grok_exporter (https://github.com/fstab/grok_exporter) - works but I can extract only limited information + one instance can monitor only one log file which mean I would have to start more instances exporting on more ports and configure all off them in prometheus - which makes too many new points of failure
  • fluentd prometheus exporter (https://github.com/kazegusuri/fluent-plugin-prometheus) - works but looks like I can extract only very simple metrics and I cannot make any advanced regexp analysis of a line(s) from log file

Does any one here has a really running solution for monitoring advanced metrics from log files using "some exporter" + Prometheus + Grafana? Or instead of exporter some program from which I could grab results using Prometheus push gateway. Thanks.

like image 878
JosMac Avatar asked Dec 15 '16 09:12

JosMac


People also ask

How do I monitor logs in Prometheus?

The log files are matches line by line using a regular expression, and the matched data is stored in different variables. Then, the data of these variables are exported via grok_exporter. Once you add the grok_exporter server to Prometheus as a target, Prometheus can monitor the desired log files via grok_exporter.

Which exporters can be used to get system metrics for Prometheus?

If you use Prometheus and Grafana for metrics storage and data visualization, Solr includes a Prometheus exporter to collect metrics and other data.

How do I monitor Grafana logs?

After login into the Grafana, click on the Explore tab. Select data source as Loki and enter the {app=”nginx”} in Log labels. Note that metadata of Nginx deployment contains the label app=nginx which is the same that filters out logs in Loki. Alerts enter into the Pending state as soon as its condition satisfies.


2 Answers

Take a look at Telegraf. It does support tailing logs using input plugins logparser and tail. To export metrics as prometheus endpoint use prometheus_client output plugin. You also may apply on the fly aggregations. I've found it simpler to configure for multiple log files than grok_exporter or mtail

like image 126
LazyS Avatar answered Sep 28 '22 09:09

LazyS


Those are the 3 answers currently for getting log data into Prometheus.

You could also look into getting whatever is producing the logs to expose Prometheus metrics directly.

like image 42
brian-brazil Avatar answered Sep 28 '22 09:09

brian-brazil