How to get container's disk usage in Kubernetes (without docker command)?

Question

I have a Kubernetes Cluster and want to know how much disk space my containers use. I am not talking about mounted Volumes.

I can get this information by using docker commands like docker system df -v or docker ps -s, but I don't want to connect to every single worker node.

Is there a way to get a container's disk usage via kubectl or are there kubelet metrics where I can get this information from?

Answer 1

Yes, but currently not with kubectl, you can get metrics from the kubelet, either through the kube-apiserver (proxied) or directly calling the kubelet HTTP(s) server endpoint (default port 10250). Disk metrics are generally available on the /stats/summary endpoint and you can also find some cAdvisor metrics on the /metrics/cavisor endpoint.

For example, to get the 'usedBytes' for the first container in the first pod returned through the kube-apiserver:

$ curl -k -s -H 'Authorization: Bearer <REDACTED>' \
  https://kube-apiserver:6443/api/v1/nodes/<node-name>/proxy/stats/summary \
  | jq '.pods[0].containers[0].rootfs.usedBytes'

The Bearer token can be a service account token tied to a ClusterRole like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
  name: myrole
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- nonResourceURLs:
  - /metrics
  - /api/*
  verbs:
  - get