How can I drop a specific label in a specific Prometheus ServiceMonitor metric using relabeling/metricRelabeling?

Question

For example, I have two metrics with different labels:

node_metrics_first{foo="bar",AAA="aaa"}  
node_metrics_second{BBB="bbb",CCC="ccc"}

How can I use relabeling/metricRelabeling drop/labeldrop in servicemonitor to remove foo label in node_metrics_first, which means I should get the result?:

node_metrics_first{AAA="aaa"}  
node_metrics_second{BBB="bbb",CCC="ccc"}

Answer 1

In Prometheus operator ServiceMonitors, you can use spec.endpoints[*].relabelings to alter labels and metrics:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
spec:
  endpoints:
  - interval: 30s
    port: metrics
    scheme: http
    metricRelabelings:   <-- here you shine!
    - action: labeldrop
      regex: (foo|otherlabeltodrop)

Answer 2

There is an extensive blog from Grafana about how the relabeling works. In your case:

  - job_name: some_job
    metric_relabel_configs:
      - regex: "foo"
        action: labeldrop