How to execute a query with two metrics in Prometheus?

Question

I am using Prometheus to query metrics from Apache Flink. I want to measure the number of records In and Out per second of a Map function. When I query two different metrics in Prometheus, the chart only shows one of them.

flink_taskmanager_job_task_operator_numRecordsInPerSecond{operator_name="Map"} 
or flink_taskmanager_job_task_operator_numRecordsOutPerSecond{operator_name="Map"}

Does not matter if I change the operator or to and. The chart shows only the first (flink_taskmanager_job_task_operator_numRecordsInPerSecond). I also have tried to edit the Prometheus config file /etc/prometheus/prometheus.yml but I don't have too much experience on Prometheus and there is something wrong in my configuration. I was basing my solution on this post.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']   
  - job_name: 'flink'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9250', 'localhost:9251', '192.168.56.20:9250']
    metrics_path: /
# HOW TO ADD THE OPERATOR NAME ON THE METRIC NAME?
    metric_relabel_configs:
      - source_labels: [__name__]
      regex: '(flink_taskmanager_job_task_operator)_(\w+)'
      replacement: '${2}'
      target_label: pool
      - source_labels: [__name__]
      regex: '(flink_taskmanager_job_task_operator)_(\w+)'
      replacement: '${1}_bytes'
      target_label: __name__

Answer 1

First of all, for more complex graphing you should definitely investigate Grafana. The built-in Prometheus graphs are useful eg. for debugging, but definitely more limited. In particular one graph will only display the results of one query.

Now for a hack that I definitely do not recommend:

flink_taskmanager_job_task_operator_numRecordsInPerSecond{operator_name="Map"}
or
label_replace(flink_taskmanager_job_task_operator_numRecordsOutPerSecond{operator_name="Map"}, "distinct", "foo", "job", ".*")

Since, as documented

vector1 or vector2 results in a vector that contains all original elements (label sets + values) of vector1 and additionally all elements of vector2 which do not have matching label sets in vector1.

you can add a new label that is not present in the labels on the first vector to the second vector and thus keep all elements from both.

Answer 2

It is possible to select multiple metric names with a single PromQL query by using a regular expression filter on __name__ label:

{__name__=~"flink_taskmanager_job_task_operator_numRecords(In|Out)PerSecond",operator_name="Map"}

See docs about the __name__ label here.

There is another solution when using Prometheus-compatible query engine such as MetricsQL by using union function:

union(
 
 flink_taskmanager_job_task_operator_numRecordsInPerSecond{operator_name="Map"},
 
 flink_taskmanager_job_task_operator_numRecordsOutPerSecond{operator_name="Map"}
)

Note that selecting multiple time series via __name__ regexp can result in vector cannot contain metrics with the same labelset error if the selected series are wrapped in any PromQL function. For example:

max_over_time(
 
 {__name__=~"flink_taskmanager_job_task_operator_numRecords(In|Out)PerSecond",operator_name="Map"}[5m]
)

This is because Prometheus removes metric names from input series when applying PromQL functions. MetricsQL from VictoriaMetrics provides a solution for this issue - keep_metric_names modifier (see these docs for details):

max_over_time(
 
 {__name__=~"flink_taskmanager_job_task_operator_numRecords(In|Out)PerSecond",operator_name="Map"}[5m]
)
keep_metric_names

P.S. I work on VictoriaMetrics and MetricsQL.