Docker: Copy file out of container while building it

Question

I build the following image with docker build -t mylambda . I now try to export lambdatest.zip to my localhost while building it so I see the .zip file on my Desktop. So far I used docker cp <Container ID>:/var/task/lambdatest.zip ~/Desktop but that doesn't work inside my Dockerfile (?). Do you have any ideas?

FROM lambci/lambda:build-python3.7
COPY lambda_function.py .

RUN python3 -m venv venv
RUN . venv/bin/activate

# ZIP
RUN pushd /var/task/venv/lib/python3.7/site-packages/
# Execute "zip" in bash for explanation of -9qr
RUN zip -9qr /var/task/lambdatest.zip *

Dockerfile (updated):

FROM lambci/lambda:build-python3.7
RUN python3 -m venv venv
RUN . venv/bin/activate

RUN pip install --upgrade pip
RUN pip install pystan==2.18
RUN pip install fbprophet

WORKDIR /var/task/venv/lib/python3.7/site-packages

COPY lambda_function.py .
COPY .lambdaignore .

RUN echo "Package size: $(du -sh | cut -f1)"

RUN zip -9qr lambdatest.zip *
RUN cat .lambdaignore | xargs zip -9qr /var/task/lambdatest.zip * -x

Answer 1

The typical answer is you do not. A Dockerfile does not have access to write files out to the host, by design, just as it does not have access to read arbitrary files from outside of the build context. There are various reasons for that, including security (you don't want an image build dropping a backdoor on a build host in the cloud) and reproducibility (images should not have dependencies outside of their context).

As a result, you need to take an extra step to extract contexts of an image back to the host. Typically this involves creating a container a running a docker cp command, along the lines of the following:

docker build -t your_image .
docker create --name extract your_image
docker cp extract:/path/to/files /path/on/host
docker rm extract

Or it can involve I/O pipes, where you run a tar command inside the container to package the files, and pipe that to a tar command running on the host to save the files.

docker build -t your_image
docker run --rm your_image tar -cC /path/in/container . | tar -xC /path/on/host

---

Recently, Docker has been working on buildx which is currently experimental. Using that, you can create a stage that consists of the files you want to export to the host and use the --output option to write that stage to the host rather than to an image. Your Dockerfile would then look like:

FROM lambci/lambda:build-python3.7 as build
COPY lambda_function.py .

RUN python3 -m venv venv
RUN . venv/bin/activate

# ZIP
RUN pushd /var/task/venv/lib/python3.7/site-packages/
# Execute "zip" in bash for explanation of -9qr
RUN zip -9qr /var/task/lambdatest.zip *

FROM scratch as artifact
COPY --from=build /var/task/lambdatest.zip /lambdatest.zip

FROM build as release

And then the build command to extract the zip file would look like:

docker buildx build --target=artifact --output type=local,dest=$(pwd)/out/ .

I believe buildx is still marked as experimental in the latest release, so to enable that, you need at least the following json entry in $HOME/.docker/config.json:

{ "experimental": "enabled" }

And then for all the buildx features, you will want to create a non-default builder with docker buildx create.

---

With recent versions of the docker CLI, integration to buildkit has exposed more options. Now it's no longer needed to run buildx to get access to the output flag. That means the above changes to:

docker build --target=artifact --output type=local,dest=$(pwd)/out/ .

If buildkit hasn't been enabled on your version (should be on by default in 20.10), you can enable it in your shell with:

export DOCKER_BUILDKIT=1

or for the entire host, you can make it the default with the following in /etc/docker/daemon.json:

{
  "features": {"buildkit": true }
}

And to use the daemon.json the docker engine needs to be reloaded:

systemctl reload docker