Monitoring Kubernetes PVC disk usage

Question

I'm trying to monitor Kubernetes PVC disk usage. I need the memory that is in use for Persistent Volume Claim. I found the command:

kubectl get --raw /api/v1/persistentvolumeclaims | jq

Return:

"status":{
            "phase":"Bound",
            "accessModes":[
               "ReadWriteOnce"
            ],
            "capacity":{
               "storage":"1Gi"
            }
         }

But it only brings me the full capacity of the disk, and as I said I need the used one

Does anyone know which command could return this information to me?

Answer 1

+1 to touchmarine's answer however I'd like to expand it a bit and add also my three cents.

But it only brings me the full capacity of the disk, and as I said I need the used one

PVC is an abstraction which represents a request for a storage and simply doesn't store such information as disk usage. As a higher level abstraction it doesn't care at all how the underlying storage is used by its consumer.

@touchmarine, Instead of using a Pod whose only function is to sleep and every time you need to check the disk usage you need to attach to it maually, I would propose to use something like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      volumes:
      - name: media
        persistentVolumeClaim:
          claimName: media
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: "/data"
          name: media
      - name: busybox
        image: busybox
        command: ["/bin/sh"]
        args: ["-c", "while true; do du -sh /data; sleep 10;done"]
        volumeMounts:
        - mountPath: "/data"
          name: media

It can be of course a single-container busybox Pod as in @touchmarine's example but here I decided to to show also how it can be used as a sidecar running next to nginx container within a single Pod.

As it runs a simple bash script - an infinite while loop, which prints out current disk usage to the standard output it can be read with kubectl logs without a need of using kubectl exec and attaching to the Pod:

$ kubectl logs nginx-deployment-56bb5c87f6-dqs5h busybox
20.0K   /data
20.0K   /data
20.0K   /data

I guess it can be also used more effectively to configure some sort of monitoring of disk usage.

Answer 2

I don't have a definitive anwser, but I hope this will help you. Also, I would be interested if someone has a better anwser.

Get current usage

The PersistentVolume subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed.

-- Persistent Volume | Kubernetes

As stated in the Kubernetes documentation, PV (PersistentVolume) and PVC (PersistentVolumeClaim) are abstractions over storage. As such, I do not think you can inspect PV or PVC, but you can inspect the storage medium.

To get the usage, create a debugging pod which will use your PVC, from which you will check the usage. This should work depending on your storage provider.

# volume-size-debugger.yaml
kind: Pod
apiVersion: v1
metadata:
  name: volume-size-debugger
spec:
  volumes:
  - name: debug-pv
    persistentVolumeClaim:
      claimName: <pvc-name>
  containers:
  - name: debugger
    image: busybox
    command: ["sleep", "3600"]
    volumeMounts:
    - mountPath: "/data"
      name: debug-pv

Apply the above manifest with kubectl apply -f volume-size-debugger.yaml, and run a shell inside it with kubectl exec -it volume-size-debugger sh. Inside the shell run du -sh to get the usage in a human readable format.

Monitoring

As I am sure you have noticed, this is not especially useful for monitoring. It may be useful for a one-time check from time to time, but not for monitoring or low disk space alerts.

One way to setup monitoring would be to have a similar sidecar pod like ours above and gather our metrics from there. One such example seems to be the node_exporter.

Another way would be to use CSI (Container Storage Interface). I have not used CSI and do not know enough about it to really explain more. But here are a couple of related issues and related Kubernetes documentation:

• Monitoring Kubernetes PersistentVolumes - prometheus-operator
• Volume stats missing - csi-digitalocean
• Storage Capacity | Kubernetes