Is it possible to cache multi-stage docker builds?

Question

I recently switched to multi-stage docker builds, and it doesn't appear that there's any caching on intermediate builds. I'm not sure if this is a docker limitation, something which just isn't available or whether I'm doing something wrong.

I am pulling down the final build and doing a --cache-from at the start of the new build, but it always runs the full build.

Answer 1

This appears to be a limitation of docker itself and is described under this issue - https://github.com/moby/moby/issues/34715

The workaround is to:

1. Build the intermediate stages with a --target
2. Push the intermediate images to the registry
3. Build the final image with a --target and use multiple --cache-from paths, listing all the intermediate images and the final image
4. Push the final image to the registry
5. For subsequent builds, pull the intermediate + final images down from the registry first

Answer 2

Since the previous answer was posted, there is now a solution using the BuildKit backend: https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources

This involves passing the argument --build-arg BUILDKIT_INLINE_CACHE=1 to your docker build command. You will also need to ensure BuildKit is being used by setting the environment variable DOCKER_BUILDKIT=1 (on Linux; I think BuildKit might be the default backend on Windows when using recent versions of Docker Desktop). A complete command line solution for CI might look something like:

export DOCKER_BUILDKIT=1 # Use cache from remote repository, tag as latest, keep cache metadata docker build -t yourname/yourapp:latest --cache-from yourname/yourapp:latest --build-arg BUILDKIT_INLINE_CACHE=1 . # Push new build up to remote repository replacing latest docker push yourname/yourapp:latest 

Some of the other commenters are asking about docker-compose. It works for this too, although you need to additionally specify the environment variable COMPOSE_DOCKER_CLI_BUILD=1 to ensure docker-compose uses the docker CLI (with BuildKit thanks to DOCKER_BUILDKIT=1) and then you can set BUILDKIT_INLINE_CACHE: 1 in the args: section of the build: section of your YAML file to ensure the required --build-arg is set.