How to debug docker-compose cache miss when building

Question

I'm executing the same docker-compose build command and I see that it misses the cache

Building service1
Step 1/31 : FROM node:10 as build-stage
 ---> a7366e5a78b2
Step 2/31 : WORKDIR /app
 ---> Using cache
 ---> 8a744e522376
Step 3/31 : COPY package.json yarn.lock ./
 ---> Using cache
 ---> 66c9bb64a364
Step 4/31 : RUN yarn install --production
 ---> Running in 707365c332e7
yarn install v1.21.1
..
..
..

As you can see the cache was missed, but I couldn't understand why What is the best method to debug what changed and try to figure out why

EDIT: The question is not to debug my specific problem. But how can I generally debug a problem of this kind. How can I know WHY docker-compose thinks things changed (although I'm pretty sure NOTHING changed), which files/commands/results are different?

Answer 1

how can I generally debug a problem of this kind. How can I know WHY docker-compose thinks things changed (although I'm pretty sure NOTHING changed), which files/commands/results are different

In general, as shown here:

I'm a bit bummed that I can't seem to find any way to make the Docker build more verbose

But when it comes to docker-compose, it depends on your version and option used.
moby/moby issue 30081 explains (by Sebastiaan van Stijn (thaJeztah):

Current versions of docker-compose and docker build in many (or all) cases will not share the build cache, or at least not produce the same digest.

The reason for that is that when sending the build context with docker-compose, it will use a slightly different compression (docker-compose is written in Python, whereas the docker cli is written in Go).
There may be other differences due to them being a different implementation (and language).

(that was also discussed in docker/compose issue 883)

The next release of docker compose will have an (currently opt-in) feature to make it use the actual docker cli to perform the build (by setting a COMPOSE_DOCKER_CLI_BUILD=1 environment variable). This was implemented in docker/compose#6865 (1.25.0-rc3+, Oct. 2019)

With that feature, docker compose can also use BuildKit for building images (setting the DOCKER_BUILDKIT=1 environment variable).
I would highly recommend using buildkit for your builds if possible.
When using BuildKit (requires Docker 18.09 or up, and at this point is not supported for building Windows containers), you will see a huge improvement in build-speed, and the duration taken to send the build-context to the daemon in repeated builds (buildkit uses an interactive session to send only those files that are needed during build, instead of uploading the entire build context).

So double-check first if your docker-compose uses BuildKit, and if the issue (caching not reused) persists then:

COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build

Sebastiaan added in issue 4012:

BuildKit is still opt-in (because there's no Windows support yet), but is production quality, and can be used as the default for building Linux images.

Finally, I realize that for the Azure pipelines, you (probably) don't have control over the versions of Docker and Docker Compose installed, but for your local machines, make sure to update to the latest 19.03 patch release; for example, Docker 19.03.3 and up have various improvements and fixes in the caching mechanisms of BuildKit (see, e.g., docker#373).

---

Note, in your particular case, even though this is not the main issue in your question, it would be interesting to know if the following helps:

yarnpkg/yarn/issue 749 suggests:

You wouldn't mount the Yarn cache directory. Instead, you should make sure you take advantage of Docker's image layer caching.

These are the commands I am using:

COPY package.json yarn.lock ./
RUN yarn --pure-lockfile

Then try your yarn install command, and see if docker still doesn't use its cache.

RUN yarn install --frozen-lockfile --production && yarn cache clean 

Don't forget a yarn cache clean in order to prevent the yarn cache from winding up in docker layers.

If the issue persists, switch to buildkit directly (for testing), with a buildctl build --progress=plain to see a more verbose output, and debug the caching situation.

---

Typically, a multi-stage approach, as shown here, can be useful:

FROM node:alpine
WORKDIR /usr/src/app
COPY . /usr/src/app/

# We don't need to do this cache clean, I guess it wastes time / saves space: https://github.com/yarnpkg/rfcs/pull/53
RUN set -ex; \
  yarn install --frozen-lockfile --production; \
  yarn cache clean; \
  yarn run build

FROM nginx:alpine
WORKDIR /usr/share/nginx/html
COPY --from=0 /usr/src/app/build/ /usr/share/nginx/html