I have this gauge metric "metric_awesome" from two different instances. What i want to do, is subtract instance one from instance two like so
metric_awesome{instance="one"} - metric_awesome{instance="two"}
Unfortunately the result set is empty. Has anyone experienced this?
PromQL supports the ability to join two metrics together: You can append a label set from one metric and append it to another at query time. This can be useful in Prometheus rule evaluations, since it lets you generate a new metric for a series by appending labels from another info metric.
Receive and aggregate metrics for consumption by a Prometheus server. DON'T USE THIS TOOL. If at all possible, you should expose a /metrics endpoint in your service and have Prometheus scrape it directly.
It is possible to have multiple matchers for the same label name. Vector selectors must either specify a name or at least one label matcher that does not match the empty string.
The issue here is that the labels don't match. What you want is:
metric_awesome{instance="one"} - ignoring(instance) metric_awesome{instance="two"}
If anyone searches this wanting to do this for a one-to-many subtraction, have a look at group_right
additionally to what was written before.
metric_awesome{instance="one"} - ignoring(instance) group_right metric_awesome{job="compare-instances"}
See also Prometheus dokumentation
By default Prometheus performs a - b
in the following way:
a
query.b
query.a
and b
results with identical sets of labels.See these docs for more details.
The metric_awesome{instance="one"} - metric_awesome{instance="two"}
returns empty result, because all the matching time series on the left side of -
contain instance="one"
label, while all the time series on the right side of -
contain instance="two"
label. There are no time series pairs with identical sets of labels here.
This behavior can be changed with on()
, ignoring()
, group_left()
and group_right()
modifiers.
For example, metric_awesome{instance="one"} - ignoring(instance) metric_awesom{instance="two"}
instructs to ignore instance
label during searching for time series pairs with identical labels. This may result in multiple-to-one matching, when multiple time series on one side of -
match a single time series on the another side. By default Prometheus returns error in this case. This can be fixed by adding group_left()
or group_right()
modifiers to -
operator:
metric_awesome{instance="one"}
- ignoring(instance) group_left()
metric_awesome{instance="two"}
or
metric_awesome{instance="one"}
- ignoring(instance) group_right()
metric_awesome{instance="two"}
See these docs for more details on these modifiers.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With