Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Prometheus : how do i sum by with 2 different metrics

Tags:

prometheus

I have 2 different metrics : metric_a with a field type metric_b with a field type (same one)

I'm trying to summarise a and b, of the same type. If type exists only on metric_a and not on metric_b - it should return metric_b's result. I've tried a lot of options on prometheus:

sum by (type)(metric_a{job=~"provision-dev"}) or vector(0) + sum by(type)(metric_b{job=~"provision-dev"}) or vector(0) : returns only the values from metric_a, and doesn't calculate metric_b's results.

sum by (type)(metric_a{job=~"provision-dev"}) + sum by(type)(metric_b{job=~"provision-dev"}) : returns only the values from metric_b, and doesn't calculate metric_a's results.

sum by (cluster_id)(provision_scale_out_failures{job=~"provision-dev"} + provision_scale_out_success{job=~"provision-dev"}) : well this isn't even a right query

Basically here's an example of a success :

metric_a :

  • type = type_1, sum = 5
  • type = type_2, sum = 2

metric_b :

  • type = type_1, sum = 4
  • type = type_3, sum = 3

result of the query :

  • type = type_1, sum = 9
  • type = type_2, sum = 2
  • type = type_3, sum = 3
like image 868
Dus Avatar asked Jan 07 '20 15:01

Dus


People also ask

How can we join two metrics in a Prometheus query?

PromQL supports the ability to join two metrics together: You can append a label set from one metric and append it to another at query time. This can be useful in Prometheus rule evaluations, since it lets you generate a new metric for a series by appending labels from another info metric.

Does Prometheus aggregate metrics?

Prometheus downsampling leverages continuous aggregates, one of the most popular and powerful features of TimescaleDB. Metric monitoring is a key pillar of any observability stack used to operate micro-service-based systems running on Kubernetes.

What is sum by in Prometheus?

Aggregation operators. Prometheus supports the following built-in aggregation operators that can be used to aggregate the elements of a single instant vector, resulting in a new vector of fewer elements with aggregated values: sum (calculate sum over dimensions) min (select minimum over dimensions)

What is offset in Prometheus?

The offset modifier allows changing the time offset for individual instant and range vectors in a query.


Video Answer


2 Answers

This is the expected behavior when using a binary operator: both side must have a matching label set to be taken into account.

If you want to be able to aggregate both side and get the single one, you first must get the union of different metrics using the __name__ label:

 sum by(__name__,type)(metric_a{job=~"provision-dev"}) or on(__name__) sum by(__name__,type)(metric_b{job=~"provision-dev"})

You can cascade the aggregation operator:

sum by (type) (sum by (__name__,type)(metric_a{job=~"provision-dev"}) or on(__name__) sum by(__name__,type)(metric_b{job=~"provision-dev"}))

Finally, you can also compact everything into:

sum by (type) ({__name__=~"metric_a|metric_b",job=~"provision-dev"})
like image 119
Michael Doubez Avatar answered Nov 18 '22 10:11

Michael Doubez


The following PromQL query should sum metric_a and metric_b by type:

(sum(metric_a) by (type) + sum(metric_b) by (type))
or
(sum(metric_a) by (type) unless sum(metric_b) by (type))
or
(sum(metric_b) by (type) unless sum(metric_a) by (type))

How it works:

  • The sum(metric_a) by (type) + sum(metric_b) by (type) sums time series with matching type label values on both sides of + according to matching rules
  • The sum(metric_a) by (type) unless sum(metric_b) by (type) returns sum(metric_a) by (type) results for type label values missing in sum(metric_b) by (type). See docs about unless operator.
  • The sum(metric_b) by (type) unless sum(metric_a) by (type) returns sum(metric_a) by (type) results for type label values missing in sum(metric_a) by (type).

Then results from these three queries are joined with or operator.

This query is equivalent to the query proposed by Michael: sum({__name__=~"metric_a|metric_b"}) by (type) .

P.S. This query can be simplified further when using MetricsQL:

sum(metric_a, metric_b)

This query works, since sum() function in MetricsQL accepts and sums arbitrary number of arguments.

like image 41
valyala Avatar answered Nov 18 '22 11:11

valyala