Docker how to run pip requirements.txt only if there was a change?

Question

I'm assuming that at some point in your build process, you're copying your entire application into the Docker image with COPY or ADD:

COPY . /opt/app
WORKDIR /opt/app
RUN pip install -r requirements.txt

The problem is that you're invalidating the Docker build cache every time you're copying the entire application into the image. This will also invalidate the cache for all subsequent build steps.

To prevent this, I'd suggest copying only the requirements.txt file in a separate build step before adding the entire application into the image:

COPY requirements.txt /opt/app/requirements.txt
WORKDIR /opt/app
RUN pip install -r requirements.txt
COPY . /opt/app
# continue as before...

As the requirements file itself probably changes only rarely, you'll be able to use the cached layers up until the point that you add your application code into the image.

This is directly mentioned in Docker's own "Best practices for writing Dockerfiles":

If you have multiple Dockerfile steps that use different files from your context, COPY them individually, rather than all at once. This will ensure that each step’s build cache is only invalidated (forcing the step to be re-run) if the specifically required files change.

For example:

COPY requirements.txt /tmp/
RUN pip install --requirement /tmp/requirements.txt
COPY . /tmp/

Results in fewer cache invalidations for the RUN step, than if you put the COPY . /tmp/ before it.