Why doesn't my docker-compose volume get updated with local file additions?

Question

Here's my docker-compose.yml, it contains a web service, which mounts a folder locally so that upon any code change watchmedo restarts the webapp django server. It's a reverse proxy with nginx.

version: '3'
services:
  webapp:
    build:
      context: .
      dockerfile: ./web/Dockerfile
    volumes:
      - ./web/:/usr/src/app/
      - staticfile_volume:/usr/src/app/public
    entrypoint: >
      watchmedo auto-restart --recursive 
        --pattern="*.py" --directory="." 
          gunicorn myapp.wsgi:application -- --bind 0.0.0.0:9000
  nginx:
    build:
      context: .
      dockerfile: ./nginx/Dockerfile
    depends_on:
      - webapp
    volumes:
      - staticfile_volume:/usr/src/app/public
volumes:
  staticfile_volume:

My local file setup is like:

$ tree -L 2
.
├── docker-compose.yml
├── nginx
│   ├── Dockerfile
│   └── nginx.conf
└── web
   ├── Dockerfile
   ├── manage.py
   ├── myapp
   ├── public
   └── static

But when I create a new file in web/public (the same folder mounted as a shared volume between webapp and nginx services), I don't see it from inside the running webapp container.

Yet, if I create a new file anywhere else in the web/ folder (which is also mounted as a separate volume), I see the change from inside the running webapp container.

What is causing this? And how do I change this behavior?

(I need to be able to run python manage.py collectstatic from inside the running container but output into my local hard drive's web/public so I can build the production Docker images for deploy.)

Answer 1

I can't see a problem with what you've got here. I copied your docker-compose.yml and removed only the entrypoint: part of it and it worked for me.

docker-compose.yml

version: '3'
services:
  webapp:
    build:
      context: .
      dockerfile: ./web/Dockerfile
    volumes:
      - ./web/:/usr/src/app/
      - staticfile_volume:/usr/src/app/public
  nginx:
    build:
      context: .
      dockerfile: ./nginx/Dockerfile
    depends_on:
      - webapp
    volumes:
      - staticfile_volume:/usr/src/app/public
volumes:
  staticfile_volume:

web/Dockerfile

FROM ubuntu:18.04

CMD tail -f /dev/null

nginx/Dockerfile

FROM nginx:latest

CMD tail -f /dev/null

Demo:

To elaborate on DannyB's comment above, make sure you're not creating the file on the host in web/public as this folder gets mounted over when the containers start. A docker volume works in a similar way to a standard linux mount - docker will just mount the new volume over the top of the existing directory.

If you want to run a command inside a Docker container that changes files on your host filesystem, don't use a docker volume - instead use a bind mount like you've done here: - ./web/:/usr/src/app/.

The difference between a bind mount and a docker volume is that a bind mount will mount files from your host inside your container, and will rely on those folders / files being on your host filesystem, and a docker volume will be completely managed by docker and can only be shared between containers, not with the host as well (although these files do live on the host somewhere, it's not practical to track them down and attempt to use them).

You can actually just remove your docker volume from your docker-compose file and add a bind mount for nginx and you'll start to see the behaviour you're after:

docker-compose.yml

version: '3'
services:
  webapp:
    build:
      context: .
      dockerfile: ./web/Dockerfile
    volumes:
      - ./web/:/usr/src/app/
  nginx:
    build:
      context: .
      dockerfile: ./nginx/Dockerfile
    depends_on:
      - webapp
    volumes:
      - ./web/public:/usr/src/app/public

Demo:

Answer 2

You've told Docker that staticfile_volume contains critical application data that must be preserved across runs of the container. The first time the container starts up, and the first time only, Docker will populate it from the image. If you update the image later, since the volume contains critical application data, Docker won't change it.

The easiest short-term workaround is to delete the volume. Try docker-compose down -v; docker-compose up --build. You'll need to do this whenever you change the static content.

It'd be a little easier long-term to configure the back-end application to serve its own files. Django has a django.contrib.staticfiles module to do this. Then your nginx proxy can unconditionally redirect to the backend container, and you don't have to worry about the file sharing issue.

(To see this better, take @ChrisMcKinnel's reproduction recipe and run it once. Then take the web/Dockerfile from there and COPY a file into /usr/src/app/public and re-run docker-compose up --build. You won't see the file appear unless you docker-compose down -v.)