When does Docker image cache invalidation occur?

Question

Maybe my Google Foo is not strong enough, but I can't find a definite list about when Docker images in the cache are invalidated. Specifically, I'm interested at least in these scenarios:

• Invalidation because of mtime changes vs checksum changes. Which applies when? Can it deal with different source paths (e.g. clones of the repository in different directories)?
• Invalidation because of updated base images. At which point will (security) updates of e.g. Debian bubble down to me?
• Are there any explicit APIs that a continuous integration tool can use to tell Docker which cached images can be reused and which can't (for example because of a wget foo.com/latest.gz)?

Answer 1

As of Docker 1.8, Docker no longer uses mtime to invalidate the cache (this changed in this pull request #12031).

When building an image;

• For local content (ADD myfiles /somewhere / COPY myfiles /somewhere), docker uses checksum changes to invalidate the cache
• Remote content (ADD http://example.com/foobar /somewhere), is always downloaded, but the build-cache is invalidated based on checksum changes
• RUN instructions (such as wget foo.com/latest.gz) will never invalidate the cache, unless the instruction is changed; i.e., the cache is based on the text in the instruction. If you want reproducible builds, make sure these URLs point to a specific version (wget http://example.com/package-major.minor.patch.gz)

Docker 1.9 introduced support for build-time arguments, which enable you to pass variables that can be used inside the Dockerfile so that you don't have to edit the Dockerfile to break the cache, or install a different version of the package.

For example

FROM foobar
ARG MAJOR=1
ARG MINOR=0
ARG PATCH=0
ADD http://example.com/package-$MAJOR.$MINOR.$PATCH.gz /

Will add http://example.com/package-1.0.0.gz by default, however, passing a "major", "minor" or "patch" build-time parameter can override the version to download, and will invalidate the cache;

docker build --build-arg MINOR=2 .                                           Sat Jan 16 13:22:40 2016
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ubuntu
 ---> 1c9b046c2850
Step 2 : ARG MAJOR=1
 ---> Using cache
 ---> a149d88772ba
Step 3 : ARG MINOR=0
 ---> Using cache
 ---> e3dae0189ffd
Step 4 : ARG PATCH=0
 ---> Using cache
 ---> 678d7ae33054
Step 5 : ADD http://example.com/package-$MAJOR.$MINOR.$PATCH.gz /
Get http://example.com/package-1.2.0.gz: dial tcp 127.0.0.1:80: getsockopt: connection refused

For more information about the build-cache, read the build-cache section in the documentation.

At which point will (security) updates of e.g. Debian bubble down to me?

Docker will not automatically download updated images, or update your images that are based on them. However, if you docker pull yourbaseimage, and a newer image is downloaded, then the build-cache for images based on that is invalidated, so the next build will not use the cache.

For automated builds on Docker hub, you can make sure that images are automatic rebuilt if the base-image is updated, see the documentation on automated builds