How to show custom application metrics in Prometheus captured using the golang client library from all pods running in Kubernetes

Question

I am trying to get some custom application metrics captured in golang using the prometheus client library to show up in Prometheus.

I have the following working:

• I have a go application which is exposing metrics on localhost:8080/metrics as described in this article:

  https://godoc.org/github.com/prometheus/client_golang/prometheus

• I have a kubernates minikube running which has Prometheus, Grafana and AlertManager running using the operator from this article:

  https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus

• I created a docker image for my go app, when I run it and go to localhost:8080/metrics I can see the prometheus metrics showing up in a browser.

• I use the following pod.yaml to deploy my docker image to a pod in k8s

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
  labels:
    zone: prod
    version: v1
  annotations:
   prometheus.io/scrape: 'true'
   prometheus.io/port: '8080'

spec:
   containers:
    - name: my-container
      image: name/my-app:latest
      imagePullPolicy: IfNotPresent
      ports:
      - containerPort: 8080

• If I connect to my pod using:

kubectl exec -it my-app-pod -- /bin/bash

then do wget on "localhost:8080/metrics", I can see my metrics

So far so good, here is where I am hitting a wall. I could have multiple pods running this same image. I want to expose all the images to prometheus as targets. How do I configure my pods so that they show up in prometheus so I can report on my custom metrics?

Thanks for any help offered!

Answer 1

The kubernetes_sd_config directive can be used to discover all pods with a given tag. Your Prometheus.yml config file should have something like so: - job_name: 'some-app' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_label_app] regex: python-app action: keep

The source label [__meta_kubernetes_pod_label_app] is basically using the Kubernetes api to look at pods that have a label of 'app' and whose value is captured by the regex expression, given on the line below (in this case, matching 'python-app').

Once you've done this Prometheus will automatically discover the pods you want and start scraping the metrics from your app.

Hope that helps. You can follow blog post here for more detail.

Note: it is worth mentioning that at the time of writing, kubernetes_sd_config is still in beta. Thus breaking changes to configuration may occur in future releases.