How to deal with "prometheus" and "prometheus_replica" labels?

Question

In a hierarchical federated setup of prometheus with a Pull model for the metrics, I see "prometheus" and "prometheus_replica" labels in the metrics that's captured. The system is monitoring a StatefulSet deployment of Kubernetes.

When querying or alerting I see duplicate data included due to these labels, i.e I see a metric with these labels and also without these. Effectively causing wrong counts and alerts.

I see "prometheus" and "prometheus_replica" labels used in the queries on the prometheus that pulls metrics from federated endpoint.

I use ServiceMonitor with Prometheus operator on every kube cluster. All the metrics is federated to a single different Prometheus where this problem is seen.

Is there any documentation on how these labels get generated? Are those metrics to be treated duplicate or ignored?

Answer 1

I've run into this issue as well, just adding for anyone getting to this comment the configuration options that affect this according to the documentation here https://github.com/prometheus-operator/prometheus-operator/blob/ca400fdc3edd0af0df896a338eca270e115b74d7/Documentation/api.md#prometheusspec . link to the code here https://github.com/prometheus-operator/prometheus-operator/blob/ca400fdc3edd0af0df896a338eca270e115b74d7/pkg/prometheus/promcfg.go#L95-L132

replicaExternalLabelName: Name of Prometheus external label used to denote replica name. Defaults to the value of prometheus_replica. External label will not be added when value is set to empty string ("").
prometheusExternalLabelName: Name of Prometheus external label used to denote Prometheus instance name. Defaults to the value of prometheus. External label will not be added when value is set to empty string ("").

So if you want to remove these duplicates, just set those options to empty strings in the Prometheus custom resource in your cluster.

Answer 2

I finally found these labels coming from the prometheus operator. It was added for an requirement that's unwritten in any documents. I see it doesn't work in 0.17 version. Its works in 0.23 version of operator.