Scraping metrics from every Pod in a DaemonSet

Question

We're using https://github.com/fluent/fluentd-kubernetes-daemonset to deploy Fluentd in our K8s cluster. We have 5 nodes in the cluster, which means there are 5 Fluentd pods.

Each Fluentd pod in the DaemonSet exposes Prometheus metrics on localhost:24231/metrics endpoint via fluentd prometheus plugin. I'm having trouble finding the relevant bits of documentation on how to configure Prometheus to collect those metrics from every Pod's localhost:24321/metrics endpoint.

TL;DR

• there are N pods in a DaemonSet
• each pod has prometheus metrics exposed on localhost:24321

I need to configure Prometheus so it's able to scrape those metrics somehow. Any tips on how to solve this or examples of such configurations would be much appreciated!

Answer 1

Solution in our case was using pod monitors CRD from the prometheus operator with podMetricEndpoints pointing at the right port:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: fluentd
spec:
  podMetricsEndpoints:
    - honorLabels: true
      interval: 15s
      path: /metrics
      targetPort: 24231
      scheme: http
  selector:
    matchLabels:
      app: fluentd