Docker image corruption? Remove layers?

Question

After a system restart, it looks like the docker images which were existing are corrupted.

I tried the following-

1. Rebuild a docker inside that machine - this worked. This image is running fine.
2. I pulled an image which was already existing, it said layers already exist. But this image still seems to be corrupted.

I feel that removing the image will help. When I try removing, it seems to remove only the tag. It is not removing all the layers. How can I do this?

I tried docker rmi. The image got removed.

Now, I try to pull the image again, some of the layers are already existing. I am trying to run the docker, it says oci runtime error ..... file not found.

These images are working on other machines, and was working on this machine till some hours before.

Answer 1

Warn :
Deleting some directories of /var/lib/docker/overlay2 may look appealing, may work but it is also risky. Indeed, it may create additional inconsistency/corrupted layers.
If you do that, please first make a full backup of folders that you want delete.
I did it two times in a production environment. The first time it worked, the second time it provoked a real mess (hard time and stress to fix that).
Since, I never did it again.

To fix issues with image pulling (layer not found or corrupted), here my tricks.

1 - If you may and want to wipe all your images and containers, you could do that.

Stop and remove all containers (running or not) :

docker rm $(docker stop $(docker ps -aq)) 

And in addition, use the system prune command :

docker system prune --volumes --all 

to delete :

• all stopped containers
• all networks not used by at least one container
• all volumes not used by at least one container
• all images without at least one container associated to them
• all build cache

To skip the confirmation dialog, add the -f flag.

It should solve corrupted layer issues since you restart from scratch.

2 - If it doesn't work (that is layers are still not found or corrupted), a possible strategy is ensuring that these failing layers are not used any longer during the pull.

To achieve that :

• If the problem is a parent image layer : I update the base image specified in the Dockerfile to find a base image version which the layer(s) with the issue is not used any longer. Generally I specify a little more recent version, I build/run. If if works, fine. If it still fails, I update the base image version further.
• If the problem is an image layer created in my Dockerfile steps itself : I specify the --no-cache flag in docker build.