I getting always" no space left on device" issue in minikube even system have free space , how to resolve?

Question

I'm running minikube (version: v1.10.1) with Helm 3 in a local system with ubuntu-16.04 and docker version 19.03.8. Currently, I'm getting a "no space left" issue even when the local disk has more than enough space.

How can I increase the space? This is the POD describe details:

  Normal   Scheduled  3m41s                default-scheduler  Successfully assigned default/greeter-5fb8ccb96b-zwzfc to minikube
  Warning  Failed     108s                 kubelet, minikube  Failed to pull image "12345.dkr.ecr.ap-south-1.amazonaws.com/micro:latest": rpc error: code = Unknown desc = write /var/lib/docker/tmp/GetImageBlob679392224: no space left on device
  Warning  Failed     55s (x2 over 108s)   kubelet, minikube  Error: ErrImagePull
  Warning  Failed     55s                  kubelet, minikube  Failed to pull image "12345.dkr.ecr.ap-south-1.amazonaws.com/micro:latest": rpc error: code = Unknown desc = write /var/lib/docker/tmp/GetImageBlob816503247: no space left on device
  Normal   BackOff    41s (x2 over 107s)   kubelet, minikube  Back-off pulling image "12345.dkr.ecr.ap-south-1.amazonaws.com/micro:latest"
  Warning  Failed     41s (x2 over 107s)   kubelet, minikube  Error: ImagePullBackOff
  Normal   Pulling    27s (x3 over 3m40s)  kubelet, minikube  Pulling image "12345.dkr.ecr.ap-south-1.amazonaws.com/micro:latest"

Answer 1

You need to check the available space on the Minikube Virtual Machine, not in your host OS.

You can access the Minikube VM with minikube ssh, then check both space and inodes availability (respectively df -h and df -i). Generally, the cause of lack of space is due to many docker images in the local repository.

Possible fixes are:

• Remove unused docker images (docker image prune -a)
• Increase the disk size on your Virtual Box configuration and resize the Linux partition (this can be painful to do and may cause a lot of trouble...)
• Start a new minikube VM with increased disk size (e.g., minikube start --disk-size 50000mb)