Add custom scrape endpoints in helm chart kube-prometheus-stack deployment

Question

First off a little new to using helm...

So I'm struggling to get the helm deployment of this: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack

To work the way I would like in my kubernetes cluster. I like what it has done so far but how can I make it scrape a custom endpoint? I have seen this: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus

Under the section titled: "Scraping Pod Metrics via Annotations". I have added the following annotations to the pod deployment (and then the node port service) in kubernetes:

annotations = {
 "prometheus.io/scrape" = "true"
 "prometheus.io/path"   = "/appMetrics/prometheusMetrics"
 "prometheus.io/port"   = "443"
}

However, when I look in the targets page of prometheus I don't see it there. I also don't see it in the configuration file. So that makes me think this helm chart isn't deploying the same prometheus chart.

So now the question is, how can I setup a custom scrape endpoint using the helm chart kube-prometheus-stack. From my reading this is the one I should* be using, right?

Answer 1

Try this below in your custom_values.yaml and apply it.

prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
      - job_name: your_job_name
        scrape_interval: 15s
        kubernetes_sd_configs:
        - role: pod
          namespaces:
            names:
              - your_namespace
        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: pod
        - source_labels: [__address__]
          action: replace
          regex: ([^:]+)(?::\d+)?
          replacement: ${1}:your_port
          target_label: __address__
        - source_labels: [__meta_kubernetes_pod_label_app]
          action: keep
          regex: your_pod_name

You need to replace your_job_name, your_namespace, your_port, your_pod_name to your deployment file. After I did the above metric and re-install Prometheus by helm chart, now I can see the target, and the metrics get exposed.