How to alert on metrics with a high cardinality in prometheus

Question

when trying to create an alert on high metric cardinality with the expression count by(__name__) ({__name__=~".+"}) > 50 I get the error: vector contains metrics with the same labelset after applying rule labels.

As the expression works when using it directly in prometheus, I wonder if there is an actual way to use it in an alert?

Answer 1

I think I found a solution for this issue, as I was trying it myself.

LT;DR

use this promQL expression for alerting on metric cardinality:

label_replace(count by(__name__) ({__name__=~".+"}), "name", "$1", "__name__", "(.+)") > 50

Long Version

The issue as stated in the Prometheus error message. After the metric vector is converted to a vector of the alert, no labels differ and therefore are duplicated. this means

vector A ( metric_a{label=test}, metric_b{label=test} )

is converted in

vector B ( alert_a{label=test}, alert_a{label=test}) 

and that is why you have duplicates

( caveat: that is at least my understanding) By adding a new label with the metric name itself, you create a unique label set.