How to delete cached/intermediate docker images after the cache gets invalidated

Question

I have a CI-pipeline that builds a docker image for my app for every run of the pipeline (and the pipeline is triggered by a code-push to the git repository.)

The docker image consists of several intermediate layers which progressively become very large in size. Most of the intermediate images are identical for each run, hence the caching mechanism of docker is significantly utilized.

However, the problem is that the final couple layers are different for each run, as they result from a COPY statement in dockerfile, where the built application artifacts are copied into the image. Since the artifacts are modified for every run, the already cached bottommost images will ALWAYS be invalidated. These images have a size of 800mb each.

What docker command can I use to identify (and delete) these image that gets replaced by newer images, i.e. when they get invalidated?

I would like to have my CI-pipeline to remove them at the end of the run so they don't end up dangling on the CI-server and waste a lot of disk space.

Answer 1

If I understand correctly: With every code push, CI pipeline creates new image, where new version of application is deployed. As a result, previously created image becomes outdated, so you want to remove it. To do so, you have to:

1. Get rid of all outdated containers, which where created from outdated image

• display all containers with command docker ps -a
• if still running, stop outdated containers with command docker stop [containerID]
• remove them with command docker rm [containerID]

2. Remove outdated images with command: docker rmi [imageID]

To sum up why this process is needed: you cannot remove any image, until it is used by any existing container (even stopped containers still require their images). For this reason, you should first stop and remove old containers, and then remove old images.

Detection part, and automation of deletion process should be based on image versions and container names, which CI pipeline generates while creating new images.

Edit 1

To list all images, which have no relationship to any tagged images, you can use command: docker images -f dangling=true. You can delete them with the command: docker images purge.

Just one thing to remember here: If you build an image without tagging it, the image will appear on the list of "dangling" images. You can avoid this situation by providing a tag when you build it.

Edit 2

The command for image purging has changed. Right now the proper command is:

docker image prune

Here is a link with a documentation