How get difference between 2 different prometheus metrics?

Question

Consider metric examples:

increase(application_executor_recordsWritten[20m])
increase(kafka_server_brokertopicmetrics_messagesin_total{topic="my_topic"}[20m])

If I execute those metrics separate on prometheus graph - everything works. But when try something like:

increase(application_executor_recordsWritten[20m]) -  increase(kafka_server_brokertopicmetrics_messagesin_total{topic="my_topic"}[20m])

I got No datapoints error.

1. May be it happens because application_executor_recordsWritten received for last 1 hour while kafka_server_brokertopicmetrics_messagesin_total is received for 6+ hours.

2. May it happens because those metric have different "gather settings", consider prometheus console output:

   application_executor_recordsWritten

   {app_name="app-name",exported_instance="application_111111111111111111",exported_job="application_111111111111111111",instance="XX.XXX.X.XX",job="job_name",number="1",role="executor"}

   kafka_server_brokertopicmetrics_messagesin_total

   {instance="XX.XXX.X.XX",job="job_name",topic="my_topic"}

Prometheus use something like ignore(???) keyword, but I can not figure out how does it work and how apply it for these metric.

Any ideas how to perform metrics difference? What is the correct syntax for this?

Answer 1

In PromQL, arithmetic binary operators between two metric ranges (aka vectors) are subject to vector matching: the operation is only applied on entries that have the same exact label set (name and avalue).

If there is a a difference and no values are paired, your get the infamous No data point error.

If you want to make them match, you have to

• either ignoring some labels that do not match (metric1 - ignoring(a_lone_label) metric2)
• or indicating on which label perform the match (metric1 - on(common_label_name_and_value) metric2)

In the examples you gave, it is unclear what should match. I would say instance and job; it could be:

increase(application_executor_recordsWritten[...]) - on (job,instance) increase(kafka_server_brokertopicmetrics_messagesin_total{topic="my_topic"}[...])

If you have one side of the operator containing elements that should be paired with more than one element of the other side (call one-to-many matching), you have to indicate which side of the operator (right or left) has more entries: using group_<side:rigth|left>.