Docker Buildx Cannot Pull From Local for Inherited Image

Question

I have 2 Dockerfiles on my Host Machine ( Ubuntu 20.04 ). I am running docker-ce version Docker version 19.03.12, build 48a66213fe with Experimental Features enabled. I am able to build each of them with "docker buildx" for the ARM architecture and successfully run them on my embedded Linux ARM board.

Dockerfile 1:

FROM python:3.8-slim-buster

COPY git /home/git

WORKDIR /home

RUN apt-get update -y && apt-get --no-install-recommends install build-essential pkg-config libzmq5 -y && \
    cd git && python3 setup.py install && apt remove --purge build-essential pkg-config -y && \
    apt auto-remove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/*

ADD publisher.py /home/publisher.py

Dockerfile 2:

FROM python:3.8-slim-buster

COPY git /home/git

WORKDIR /home

RUN apt-get update -y && apt-get --no-install-recommends install build-essential pkg-config libzmq5 -y && \
    cd git && python3 setup.py install && apt remove --purge build-essential pkg-config -y && \
    apt auto-remove -y && apt-get clean -y && rm -rf /var/lib/apt/lists/*

ADD subscriber.py /home/subscriber.py

Build process to create ARM compatible image on Host:

docker buildx create --name builder || true
docker buildx use builder
docker buildx build --platform linux/arm/v7 -t "company-publisher:v1.3" . --load
docker save company-publisher:v1.3 > company-publisher-v1.3.tar

Loading the image on the ARM:

docker load < ./company-publisher-v1.3.tar

The steps are same for the subsriber.

Since the images are basically the same, I wanted to change the publisher Dockerfile to the following:

FROM company-subscriber:v1.3

ADD publisher.py /home/publisher.py

Docker images shows that it is there locally:

REPOSITORY                   TAG                 IMAGE ID            CREATED             SIZE
company-subscriber          v1.3                d2002fa18a8d        9 hours ago         121MB

But I get the error shown below - It ALWAYS tries to pull from docker.io ( which obviously does not have the image I am trying to inherit from ):

docker buildx build --platform linux/arm/v7 -t "company-publisher:v1.3" . --load
[+] Building 1.5s (5/6)                                                                                                                                                                                              
 => [internal] load .dockerignore                                                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                                                 0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                            0.0s
 => => transferring dockerfile: 104B                                                                                                                                                                            0.0s
 => ERROR [internal] load metadata for docker.io/library/company-subscriber:v1.3                                                                                                                               0.8s
 => [internal] load build context                                                                                                                                                                               0.0s
 => => transferring context: 34B                                                                                                                                                                                0.0s
 => ERROR [1/2] FROM docker.io/library/company-subscriber:v1.3                                                                                                                                                 0.7s
 => => resolve docker.io/library/company-subscriber:v1.3                                                                                                                                                       0.7s
------
 > [internal] load metadata for docker.io/library/company-subscriber:v1.3:
------
------
 > [1/2] FROM docker.io/library/company-subscriber:v1.3:
------
failed to solve: rpc error: code = Unknown desc = failed to load cache key: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

How can I get buildx to work with a local image? Thanks.

Answer 1

There are a few different buildx drivers, and they each have tradeoffs.

First is the docker driver. This is the driver for the default builder instance if you change nothing else. It's build-in to to the docker engine and should have visibility to the other images on the host. The goal is to be similar to the classic build process.

The second is docker-container and is the default if you create a new builder instance with docker buildx create. This is needed for specific functionality like the multi-platform images and exporting the cache. But since it's running inside a container, you won't see the other images on the docker host.

---

One big issue when trying to use the docker host for multi-architecture images is that the docker engine itself doesn't support multi-architecture images. It will only pull one of the architectures from a registry, so your image becomes a single architecture that likely can't be used in a multi-architecture build.

The easiest fix is to use a registry for your images. This supports the multi-architecture image formats which you can't do on a docker host. And this is portable when you run the build on another node.

There are other options in the buildx documentation to cache from/to other locations. But when dealing with a multi-arch base image, you'll find the external registry is much easier, and likely the one that actually works. Keep in mind this doesn't have to be Docker Hub, you can run you own registry server on the same host where you run your builds.

---

Side note: buildx/buildkit also benefits from having a persistent volume if you happen to run ephemeral builders (e.g. using some kind of DinD on a CI server). Buildkit can be configured to automatically garbage collect this cache to avoid the storage problems of the past. And with that cache, you avoid the need to download the image layers on every build from the external registry.