How to make docker-build to cache the pip installs as well?

Question

I am working with the following Dockerfile.

FROM ubuntu

RUN apt-get update
RUN apt-get install -y \
    python3.9 \
    python3-pip

COPY . /app
WORKDIR /app

RUN pip install -r requirements.txt

EXPOSE 8501

ENTRYPOINT [ "streamlit", "run" ]
CMD ["app.py"]

Whenever, I rebuild the image, the docker uses the previously cached version of image and builds it quickly, except when it reaches the RUN pip install -r requirements.txt part, which it runs every time I rebuild the image (does not cache). The problem is that one of the requirement in the requirements.txt is streamlit==1.9.0, which has a large number of dependencies, so it slows the down the rebuild process. Long story short, the docker is cacheing RUN apt-get install foo but not the RUN pip install bar, which, I guess, is expected. How would you find a work-around it, to speed-up the image rebuild process when you have a long list in requirements.txt file?

Answer 1

It can't cache it because your files in app are changing constantly (I guess). The way its usually done is to copy requirements.txt seperately first, then install, then copy everything else. This way, docker can cache the install as requirements.txt didn't change.

FROM ubuntu

RUN apt-get update
RUN apt-get install -y \
    python3.9 \
    python3-pip

WORKDIR /app
COPY requirements.txt . # Copy ONLY requirements. This layer is cached if this file doesn't change

RUN pip install -r requirements.txt # This uses the cached layer if the previous layer was cached aswell

COPY . .

EXPOSE 8501

ENTRYPOINT [ "streamlit", "run" ]
CMD ["app.py"]

Answer 2

Docker BuildKit has recently introduced mounting during build time.

See https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md

For particular scenario for PIP, check out https://dev.doroshev.com/docker-mount-type-cache/