Divide two metrics in prometheus

Question

I have two metrics, scratched from telegraf.

first metric

vsphere_host_mem_active_average{esxhostname=~"esx1"}

gives one value

vsphere_host_mem_active_average{clustername="BCH1",collector="telegraf",dcname="DC",esxhostname="esx1",host="vm01",hostname="hostname1",instance="localhost:9273",job="vSphere",moid="host-78563",source="esx1",type="vmware",vcenter="vmc"}    17763152

second one

vsphere_vm_mem_granted_average{esxhostname=~"esx1"})

gives several with different labels

1.

vsphere_vm_mem_granted_average{clustername="BCH1",dcname="DC",esxhostname="esx1",guest="debian9_64",host="vm01",moid="vm-79139",source="vm01",uuid="42244f7b-abeb-92be-3e67-af19a9d8dfbd",vcenter="vmc",vmname="vm01"} 4.19418e+06

2.

vsphere_vm_mem_granted_average{clustername="BCH1",dcname="DC",esxhostname="esx1",guest="debian9_64",host="vm01",moid="vm-79146",source="vm01",uuid="4224ed0c-f306-202c-fc99-35e48fe52370",vcenter="vmc",vmname="vm02"} 8.377904e+06

is that possible to get result of first value divided by second values for each of set of labels?

Answer 1

Prometheus applies arithmetic operators such as /, -, +, * individually per each pair of time series with identical set of labels (ignoring metric name) on both sides of the operator. If there are no pairs of time series with identical labels, then Prometheus returns nothing. See these docs for more details. This behavior can be augmented by applying on(), ignoring(), group_left() and group_right() modifiers - see these docs.

So, if you need to divide a single time series by two other time series with distinct set of labels, then the following PromQL query should work:

vsphere_host_mem_active_average{esxhostname=~"esx1"}
  / on() group_right()
vsphere_vm_mem_granted_average{esxhostname=~"esx1"}

The on() modifier instructs Prometheus to limit the set of labels, which are used to find pairs of time series with identical labels on the left and the right side of / operator, to an empty set. The empty list at on() modifier automatically matches any time series on the left side of / operator to any time series on the right side of / operator.

The group_right() modifier instructs Prometheus to divide a single time series on the left side of / operator individually to every time series on the right side of / operator. The resulting time series contain labels from the right side time series. Additional labels from time series on the left side can be added to result by enumerating them inside group_right() modifier.