Permission denied when running `mkdir` inside of a Docker container

Question

I am using Docker Compose to run several containers, including one with a Postgres image. I am attempting to add a volume to that container to persist my data across container builds. However, I am receiving an error when it tries to create a directory for this volume within the container.

I run:

docker-compose build

then

docker-compose up

And I receive the following error:

ERROR: for cxbenchmark_db_1 Cannot start service db: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused \"rootfs_linux.go:57: mounting \\"/var/lib/docker/volumes/69845a017b4465e9122852a75ca194db473df95fa218658b8a60fb56eba9be9e/_data\\" to rootfs \\"/var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged\\" at \\"/var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data\\" caused \\"mkdir /var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data: permission denied\\"\""

My full docker-compose.yml looks like this (note the service called db where the volume is defined):

version: '3'  

services:  
  nginx:
    image: nginx:latest
    ports:
      - 80:8000
    volumes:
      - ./src:/src
      - ./config/nginx:/etc/nginx/conf.d
      - ./src/static:/static
    depends_on:
      - web

  web:
    build: .
    command: bash -c "python manage.py makemigrations && python manage.py migrate && gunicorn cx_benchmark.wsgi -b 0.0.0.0:8000"
    depends_on:
      - db
    volumes:
      - ./src:/src
      - ./src/static:/static
    expose:
      - 8000

  db:
    image: postgres:latest
    volumes:
      - /private/var/lib/postgresql:/var/lib/postgresql
    ports:
      - 5432:5432

Any ideas for how to solve?

Answer 1

The error you are seeing is not a problem (necessarily) with the explicit volume bind mount in your compose file, but rather with the VOLUME declaration in the main postgres official Docker image Dockerfile:

VOLUME /var/lib/postgresql/data

Since you haven't provided a mount-point for this directory (but rather the parent), the docker engine is creating a local volume and then trying to mount that volume into your already bind-mounted location and getting a permissions error.

For clarity, here is the volume the docker engine created for you:

/var/lib/docker/volumes/69845a017b4465e9122852a75ca194db473df95fa218658b8a60fb56eba9be9e/_data

And here is the directory location at which it is trying to bind mount that dir; on top of your bind mount from /private/var/lib/postgresql:

mkdir /var/lib/docker/overlay2/627956d63fb0480448079577a83b0b54f83866fdf31136b7c669541c3f672355/merged/var/lib/postgresql/data: permission denied

Now, I think the reason this is failing is that you may have turned on user namespaces in your Docker engine ("userns-remap" flag/setting) such that the container doesn't have permissions to create a directory in that root-owned location on your host. Barring that, the only other option is that the postgres container is starting as a non-root user, but I don't see anything in your compose file or the official Dockerfile for the latest release that uses the USER directive.

As an aside, since you are ending up with double-volumes because your bind mount doesn't match the VOLUME specifier in the postgres Dockerfile, you could change your compose file to mount to /var/lib/postgresql/data and get around that extra volume being created. Especially if you expect your DB data to end up in /private/var/lib/postgresql, as it may be surprising to find it isn't there, but rather in the /var/lib/docker/volumes/.. location.