Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Filter prometheus results by metric value, not by label value

Because Prometheus topk returns more results than expected, and because https://github.com/prometheus/prometheus/issues/586 requires client-side processing that has not yet been made available via https://github.com/grafana/grafana/issues/7664, I'm trying to pursue a different near-term work-around to my similar problem.

In my particular case most of the metric values that I want to graph will be zero most of the time. Only when they are above zero are they interesting.

I can find ways to write prometheus queries to filter data points based on the value of a label, but I haven't yet been able to find a way to tell prometheus to return time series data points only if the value of the metric meets a certain condition. In my case, I want to filter for a value greater than zero.

Can I add a condition to a prometheus query that filters data points based on the metric value? If so, where can I find an example of the syntax to do that?

like image 360
Steve Dwire Avatar asked Oct 11 '17 21:10

Steve Dwire


People also ask

How do you use the wildcard in Prometheus query?

Wildcards in queries Prometheus uses a Regex-like pattern and the wildcard character is . + (read: dot plus) combined with tilde character ( ~ ) instead of just the equal sign ( = ). So the query becomes http_status{job~="customer-. +"} .

What is a PromQL query?

PromQL, short for Prometheus Querying Language, is the main way to query metrics within Prometheus. You can display an expression's return either as a graph or export it using the HTTP API. PromQL uses three data types: scalars, range vectors, and instant vectors. It also uses strings, but only as literals.

Which query language does Prometheus use to filter the data?

Prometheus provides a functional query language called PromQL (Prometheus Query Language) that lets the user select and aggregate time series data in real time.

What are the 4 metric types Prometheus support?

There are four primary Prometheus metric types to query for via PromQL: counters, gauges, histograms, and summaries. These metric types fulfill the needs or requirements for most use cases, and are found in Prometheus' official client libraries: Go, Java, Ruby, and Python.


2 Answers

If you're confused by brian's answer: The result of filtering with a comparison operator is not a boolean, but the filtered series. E.g.

min(flink_rocksdb_actual_delayed_write_rate > 0)

Will show the minimum value above 0.

In case you actually want a boolean (or rather 0 or 1), use something like

sum (flink_rocksdb_actual_delayed_write_rate >bool 0)

which will give you the non-zero count.

like image 53
Caesar Avatar answered Oct 07 '22 03:10

Caesar


Filtering is done with the comparison operators, for example x > 0.

like image 34
brian-brazil Avatar answered Oct 07 '22 02:10

brian-brazil