How to set a label in Service monitor so it appears in Prometheus metrics?

Question

I want to set targetLabels in Service monitor like key value map so it appears in the Prometheus metrics. I tried with relabeling in service monitor but it didnt work.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-application
  labels:
    team: backend-team
spec:
  selector:
    matchLabels:
      app: example-application
  endpoints:
  - port: backend
    relabelings:
    - sourceLabels: [__name__]
      regex: (.*)
      targetLabel: teamname
      replacement: "backend-team"
      action: replace

I also found there is targetLabels.But not sure how to use it. TargetLabels transfers labels on the Kubernetes Service onto the target.

I also read there is whitelisting labels possible in service monitor. Please let me know how to do this and which method would be correct option..

Answer 1

In Servicemonitor spec,we need to add targetlabels in order to propagate the service labels to Prometheus.

Example service with "teamname" label:

kind: Service
apiVersion: v1
metadata:
  name: example-application
  labels:
    app: example-application
    teamname: neon
spec:
  selector:
    app: example-application
  ports:
  - name: backend
    port: 8080

Example Servicemonitor propagating the "teamname" label from service:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-application
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: example-application
  endpoints:
  - port: backend
    path: /prometheus
  namespaceSelector:
    matchNames:
    - testns
  targetLabels:
    - teamname
    - app