Caching Docker images in CircleCI

Question

I am using CircleCI to run tests and deploy a Docker image, but I'm having trouble caching Docker images, leading to very long build times.

Here's a minimal example that does not work:

circle.yml

machine:
   services:
      - docker

dependencies:
    cache_directories:
      - "~/docker"
    override:
      - if [[ -e ~/docker/image.tar ]]; then echo "cached files exists!" ; docker load -i ~/docker/image.tar; fi
      - docker images
      - docker build -t myorg/myapp:v1.1.1 .
      - mkdir -p ~/docker; docker save myorg/myapp:v1.1.1 > ~/docker/image.tar

Dockerfile:

FROM debian:latest
MAINTAINER My Name <[email protected]>
RUN apt-get update && apt-get install -y vim
CMD ["sleep", "3"]

When pushing to GitHub, the build runs fine in CircleCI. But the Docker image is rebuilt every time it runs. This gets very time consuming in a production Docker image.

The file image.tar is clearly loaded, which is clear from the output of the docker images command, but it still builds the total image every time, even when the Dockerfile does not change.

Here's some of the output from the line docker build -t myorg/myapp:v1.1.1 .

docker build -t myorg/myapp:v1.1.1 .
Sending build context to Docker daemon 67.07 kB


Step 1 : FROM debian:latest
latest: Pulling from library/debian


Digest: sha256:370807fef6f790d8519399026d26461bdf8360f94ab450da94c2350bea3cc66d
Status: Downloaded newer image for debian:latest
 ---> ae85c48b369c
Step 2 : MAINTAINER My Name <[email protected]>
 ---> Running in 4ffd69c2a82d
 ---> 6029750ba0f3
Error removing intermediate container 4ffd69c2a82d: rmdriverfs: Driver btrfs failed to remove root filesystem 4ffd69c2a82d644ea7ee8576cc06e67ee412f651edecbc40932394c057aa931d: Failed to destroy btrfs snapshot /var/lib/docker/btrfs/subvolumes for 4ffd69c2a82d644ea7ee8576cc06e67ee412f651edecbc40932394c057aa931d: operation not permitted
Step 3 : RUN apt-get update && apt-get install -y vim
 ---> Running in 544047b8c170
Ign http://httpredir.debian.org jessie InRelease
Get:1 http://httpredir.debian.org jessie-updates InRelease [142 kB]
Get:2 http://httpredir.debian.org jessie Release.gpg [2373 B]
Get:3 http://httpredir.debian.org jessie Release [148 kB]
Get:4 http://security.debian.org jessie/updates InRelease [63.1 kB]
Get:5 http://httpredir.debian.org jessie-updates/main amd64 Packages [17.6 kB]
Get:6 http://httpredir.debian.org jessie/main amd64 Packages [9064 kB]
Get:7 http://security.debian.org jessie/updates/main amd64 Packages [390 kB]
Fetched 9828 kB in 2s (4030 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
The following extra packages will be installed:
  libgpm2 vim-common vim-runtime
Suggested packages:
  gpm ctags vim-doc vim-scripts
The following NEW packages will be installed:
  libgpm2 vim vim-common vim-runtime
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 6218 kB of archives.
After this operation, 28.9 MB of additional disk space will be used.
Get:1 http://httpredir.debian.org/debian/ jessie/main libgpm2 amd64 1.20.4-6.1+b2 [34.0 kB]
Get:2 http://httpredir.debian.org/debian/ jessie/main vim-common amd64 2:7.4.488-7 [184 kB]
Get:3 http://httpredir.debian.org/debian/ jessie/main vim-runtime all 2:7.4.488-7 [5047 kB]
Get:4 http://httpredir.debian.org/debian/ jessie/main vim amd64 2:7.4.488-7 [953 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 6218 kB in 0s (10.7 MB/s)
Selecting previously unselected package libgpm2:amd64.

Any idea why it is rebuilding the docker image every time?

Answer 1

Now you can enable docker layer caching on CircleCI. It is a premium feature. It will reuse previous layers used to build your images.

https://circleci.com/docs/2.0/docker-layer-caching/