Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Prometheus - exclude 0 values from query result

I'm displaying Prometheus query on a Grafana table. That's the query (Counter metric):

sum(increase(check_fail{app="monitor"}[20m])) by (reason)

The result is a table of failure reason and its count.
The problem is that the table is also showing reasons that happened 0 times in the time frame and I don't want to display them.
AFAIK it's not possible to hide them through Grafana.

I know prometheus has comparison operators but I wasn't able to apply them.

like image 415
nirsky Avatar asked Feb 19 '19 08:02

nirsky


People also ask

What is NaN in Prometheus?

NaN is just a number in Prometheus. Some monitoring systems use NaN as a null or missing value, however in Prometheus NaN is just another floating point value. The way Prometheus represents missing data is to have the data, uhm, missing.

How do you use the wildcard in Prometheus query?

Prometheus uses the tilde character ~ to indicate a query contains a wildcard. Inside the label-query the "dot plus" ( . + ) character combination is used where all characters are accepted.

How do I filter Prometheus metrics?

To filter out unwanted metrics from a target, use the ignore_metrics configuration option. To filter out targets instead of the metrics, use the scrape_enabled_label configuration option.

What is offset in Prometheus?

The offset modifier allows changing the time offset for individual instant and range vectors in a query. For example, the following expression returns the value of http_requests_total 5 minutes in the past relative to the current query evaluation time: http_requests_total offset 5m.


1 Answers

I don't know how you tried to apply the comparison operators, but if I use this very similar query:

sum(increase(up[1d])) by (job)

I get a result of zero for all jobs that have not restarted over the past day and a non-zero result for jobs that have had instances restart.

If I now tack on a != 0 to the end of it, all zero values are filtered out:

sum(increase(up[1d])) by (job) != 0
like image 160
Alin Sînpălean Avatar answered Sep 20 '22 14:09

Alin Sînpălean