Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is it possible to cache multi-stage docker builds?

I recently switched to multi-stage docker builds, and it doesn't appear that there's any caching on intermediate builds. I'm not sure if this is a docker limitation, something which just isn't available or whether I'm doing something wrong.

I am pulling down the final build and doing a --cache-from at the start of the new build, but it always runs the full build.

like image 219
Matthew Goslett Avatar asked Oct 04 '18 12:10

Matthew Goslett


People also ask

Does Docker build cache?

Docker's build-cache is a handy feature. It speeds up Docker builds due to reusing previously created layers. You can use the --no-cache option to disable caching or use a custom Docker build argument to enforce rebuilding from a certain step.

Are Docker layers cached?

Each layer contains the filesystem changes to the image for the state before the execution of the command and the state after the execution of the command. Docker uses a layer cache to optimize and speed up the process of building Docker images.

Can you have multiple Dockerfiles?

Let's say we have two Dockerfiles, one for building the backend and another for building the frontend. We can name them appropriately and invoke the build command two times, each time passing the name of one of the Dockerfiles: $ docker build -f Dockerfile.

What is the purpose of multi stage builds in Docker?

A multistage build allows you to use multiple images to build a final product. In a multistage build, you have a single Dockerfile, but can define multiple images inside it to help build the final image.


2 Answers

This appears to be a limitation of docker itself and is described under this issue - https://github.com/moby/moby/issues/34715

The workaround is to:

  1. Build the intermediate stages with a --target
  2. Push the intermediate images to the registry
  3. Build the final image with a --target and use multiple --cache-from paths, listing all the intermediate images and the final image
  4. Push the final image to the registry
  5. For subsequent builds, pull the intermediate + final images down from the registry first
like image 72
Matthew Goslett Avatar answered Sep 21 '22 20:09

Matthew Goslett


Since the previous answer was posted, there is now a solution using the BuildKit backend: https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources

This involves passing the argument --build-arg BUILDKIT_INLINE_CACHE=1 to your docker build command. You will also need to ensure BuildKit is being used by setting the environment variable DOCKER_BUILDKIT=1 (on Linux; I think BuildKit might be the default backend on Windows when using recent versions of Docker Desktop). A complete command line solution for CI might look something like:

export DOCKER_BUILDKIT=1 # Use cache from remote repository, tag as latest, keep cache metadata docker build -t yourname/yourapp:latest --cache-from yourname/yourapp:latest --build-arg BUILDKIT_INLINE_CACHE=1 . # Push new build up to remote repository replacing latest docker push yourname/yourapp:latest 

Some of the other commenters are asking about docker-compose. It works for this too, although you need to additionally specify the environment variable COMPOSE_DOCKER_CLI_BUILD=1 to ensure docker-compose uses the docker CLI (with BuildKit thanks to DOCKER_BUILDKIT=1) and then you can set BUILDKIT_INLINE_CACHE: 1 in the args: section of the build: section of your YAML file to ensure the required --build-arg is set.

like image 25
Eric Greveson Avatar answered Sep 19 '22 20:09

Eric Greveson