how to solve the disk pressure in kubernetes

Question

I have a local OpenNESS network edge cluster using Kubernetes as its infrastructure management. I'm facing the disk pressure issue due to which pods are getting Evicted and in CrashLoopBack state. Also, the images from worker-node went missing(got deleted automatically) If I check the disk usage, I see 83% been used by the dev/sda4 or overlay filesystem. how to solve this issue.

image attached shows the disk usage

Answer 1

Your disk usage chart reveals a lot of disk usage on the overlay filesystem, so by Docker containers union file system. This suggests that you are having some large containers running. Those might have been large to start with or be writing binary data to the container file system while running.

To get to the bottom of this, you can either have a look into at your monitoring (if present). Or, your can ssh into the affected node and try to identify the "guilty" pod with:

du --max-depth=1 /var/lib/docker/overlay2/ | sort -n

and a subsequent: du | sort -n in the biggest folder.