How to scrape metrics-server to prometheus outside kubernetes cluster

Question

I have installed kubernetes metrics-server https://github.com/kubernetes-sigs/metrics-server It works fine, I can use "kubectl top pods" & "kubectl top nodes", Now I want to scrape the metrics to prometheus outside kubernetes cluster to be shown on Grafana, What I have tried are

1. Expose the metrics-server service using NodePort but not working, some https issue appear
2. Install metrics-server-exporter from https://github.com/ghouscht/metrics-server-exporter/ (this one only scrape few metrics, I cannot see it on grafana) and https://github.com/grupozap/metrics-server-exporter/ (this one is not an http application, so I cannot expose as a NodePort Service)

What should I do?

Answer 1

Try these out and let me know.

1.First you should run kube-state-metrics which collects all kubernetes metrics .

2.Using pod annotations on the kube-state-metrics , expose metrics like

prometheus.io/scrape: 'true'
prometheus.io/port: 'port'
prometheus.io/path: '/metrics'
prometheus.io/scheme: 'http'

This should expose your metrics from the pod level . To check this you can exec in to the pod and do a curl request on the port followed by the path

3.Now run a prometheus agent which can scrape metrics from this port and send it to the backend db server

4.Configure prometheus ds in your grafana agent and it should be done

Answer 2

You can use helm to install kube-prometheus-stack.

It will install Node Exporter, kube-state-metrics and prometheus operator along with other dependencies to allow you to monitor your cluster.

You can install it by using:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add stable https://charts.helm.sh/stable
helm repo update
helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack

Once it is installed you can visit your grafana URL and configure dashboard.

You can add prometheus as data source [Configuration -> Data Source] in grafana (prometheus URL should be set by default) and import kubernetes dashboard to show you summary metrics about containers running on Kubernetes nodes.