How to monitor disk usage of kubernetes persistent volumes?

Question

I have container_fs_usage_bytes with prometheus to monitor container root fs, but it seems that there is no metrics for other volumes in cAdvisor.

Answer 1

I confirmed that Kubernetes 1.8 expose metrics for prometheus.

• kubelet_volume_stats_available_bytes
• kubelet_volume_stats_capacity_bytes
• kubelet_volume_stats_inodes
• kubelet_volume_stats_inodes_free
• kubelet_volume_stats_inodes_used
• kubelet_volume_stats_used_bytes

Answer 2

Metrics for volumes are available via the kubelet summary API (/stats/summary). However, each volume plugin has to implement their own metrics. As of Kubernetes 1.7, the current volume plugins that have implemented metrics include: emptydir, secrets, gce pd, aws ebs, azure file, flocker, and portworx

Answer 3

The following metrics must be used for monitoring persistent volume stats in Kubernetes (the PVC name is exported in persistentvolumeclaim label):

• kubelet_volume_stats_capacity_bytes - the per-PVC capacity in bytes.
• kubelet_volume_stats_used_bytes - the per-PVC space usage in bytes.
• kubelet_volume_stats_available_bytes - the per-PVC free space in bytes.

The following PromQL queries can be used for determining per-pod PVC disk space usage in bytes:

sum(kubelet_volume_stats_used_bytes) by (namespace,persistentvolumeclaim)
  * on(namespace,persistentvolumeclaim) group_left(pod)
kube_pod_spec_volumes_persistentvolumeclaims_info

The following query can be used for determining free PVC disk space in bytes per each pod:

sum(kubelet_volume_stats_available_bytes) by (namespace,persistentvolumeclaim)
  * on(namespace,persistentvolumeclaim) group_left(pod)
kube_pod_spec_volumes_persistentvolumeclaims_info