Why isn't docker reusing docker-compose's cache layers?

Question

This is a cut-down example of a problem I'm having with a bigger Dockerfile.

Here's a Dockerfile:

FROM alpine:latest AS base
COPY docker-compose.yml /tmp/docker-compose.yml
RUN touch /tmp/foo

Here's a docker-compose.yml:

version: '3.5'
services:
  web:
    build:
      context: .

What I expect is that docker build will be able to reuse the cached layers that docker-compose builds. What I see when I run docker-compose build web is:

  $ docker-compose build web
Building web
Step 1/3 : FROM alpine:latest AS base
 ---> f70734b6a266
Step 2/3 : COPY docker-compose.yml /tmp/docker-compose.yml
 ---> 764c54eb3dd4
Step 3/3 : RUN touch /tmp/foo
 ---> Running in 77bdf96af899
Removing intermediate container 77bdf96af899
 ---> 7d8197f7004f

Successfully built 7d8197f7004f
Successfully tagged docker-compose-caching_web:latest

If I re-run docker-compose build web, I get:

...
Step 2/3 : COPY docker-compose.yml /tmp/docker-compose.yml
 ---> Using cache
 ---> 764c54eb3dd4
...

So it's clearly able to cache the layer with the file in it. However, when I run docker build ., here's the output I see:

  $ docker build .
Sending build context to Docker daemon  3.072kB
Step 1/3 : FROM alpine:latest AS base
 ---> f70734b6a266
Step 2/3 : COPY docker-compose.yml /tmp/docker-compose.yml
 ---> e8679333ba0d
Step 3/3 : RUN touch /tmp/foo
 ---> Running in af26cc65312d
Removing intermediate container af26cc65312d
 ---> 186c8341ee96
Successfully built 186c8341ee96

Note step 2 didn't come from the cache. Why not? Or, more importantly, how can I ensure that it does without using --cache-from?

The problem this causes is that after this step in my bigger Dockerfile that I'm not showing, there's a honking great RUN command that takes an age to run. How can I get docker build and docker-compose build to share cache layers?

(Docker Desktop v 2.3.0.2 (45183) on OS X 10.14.6 for those playing along at home)

Answer 1

With Docker-compose 1.25+ (Dec. 2019), try and use:

COMPOSE_DOCKER_CLI_BUILD=1 docker-compose build

That is what is needed to enable the docker-cli, instead of the own internal docker-compose build.

See also "Faster builds in Docker Compose 1.25.1 thanks to BuildKit Support".
But be aware of docker-compose issue 7336, when using it with DOCKER_BUILDKIT=1 (in addition of COMPOSE_DOCKER_CLI_BUILD=1)

Answer 2

Looks like a known issue. For reasons I don't entirely understand, hashes generated by docker compose build are different from those generated by docker build.

https://github.com/docker/compose/issues/883