How to scrape pod level info using prometheus kubernetes?

Question

I am trying to scrape pod level info using prometheus kubernetes. Here is the config i am using:

 - job_name: 'kubernetes-pods'

  kubernetes_sd_configs:
  - api_servers:
    - 'https://kubernetes.default'
    role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_pod_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

But i don't see any info on grafana. Do I need to make any changes in my apps? snapshot

Answer 1

With that configuration the first action asks that the pod be annotated with prometheus.io/scrape=true. Have you set that annotation on the pods in question?

Answer 2

Be aware,

there is a configuration mistake here: __meta_kubernetes_pod_namespace does not exist and should be __meta_kubernetes_namespace

Best regards,

Bart