Is there a way to create Grafana template variable using instant query?

Question

My use case is to query labels only for non-absent metric.

label_values(memory_usage,name)

The problem is the result will contain 'name' labels for metrics which are absent (metrics comes from cadvisor, so stopped containers' metric will be absent but present as a timeseries).

On prometheus side I can solve this by invoking memory_usage on api/v1/query instead of api/v1/query_range,grafana handle this as well but only on the dashbaord panels where we have an 'instant' checkbox.

But how can I do this when defining template variable?

So I need something like this:

label_values(memory_usage *if memory usage non-absent for label name*,name)

or

label_values(memory_usage *instant*,name)

Answer 1

Yes, this is possible.

As you mention, using the label_values(...) function in the variable query field does ranged queries. But as explained here, you can use query_result(query) to perform an instant query.

This will return a list of metrics, but cannot be combined with label_values(...). Using the regex field, you can extract labels from these metrics.

For example:

• Query: query_result(kafka_controller_kafkacontroller_activecontrollercount_value{} )
• Regex: /.*instance="(?<value>[\d\.:]+)".*/
• Resulting values: 10.244.120.121:5556, 10.244.120.87:5556, 10.244.120.97:5556

For reference, the output using label_values:

• Query: label_values(kafka_controller_kafkacontroller_activecontrollercount_value{}, instance)
• Regex: none
• Resulting values: 10.244.120.121:5556, 10.244.120.87:5556, 10.244.120.93:5556, 10.244.120.97:5556 (this includes a stopped pod)