Kubernetes prometheus metrics for running pods and nodes?

Question

I've set up prometheus to monitor kubernetes metrics by following the prometheus documentation.

A lot of useful metrics now show up in prometheus.

However, I can't see any metrics referencing the status of my pods or nodes.

Ideally - I'd like to be able to graph the pod status (Running, Pending, CrashLoopBackOff, Error) and nodes (NodeReady, Ready).

Is this metric anywhere? If not, can I add it somewhere? And how?

Answer 1

The regular kubernetes setup does not expose these metrics - further discussion here.

However, another service can be used to collect these cluster level metrics: https://github.com/kubernetes/kube-state-metrics.

This currently provides node_status_ready and pod_container_restarts which sound like what I want.