Mount volume for remote Docker context via SSH

Question

I'm deploying few Docker services via docker-compose with remote context. I configured it to use SSH:

docker context create remote --docker "host=ssh://[email protected]"
docker context use remote

On the remote host I have multiple configuration files which I want to mount into the Docker. It's working fine when I'm trying with docker CLI:

docker run -v /home/user/run:/test -it alpine:3.11

# ls -la /test
-> shows remote files correctly here

But when I'm starting it using docker-compose with config file:

version: "3.3"
services:
  nginx:
    image: nginx:1.17.10-alpine
    container_name: nginx
    restart: unless-stopped
    volumes:
      - ${HOME}/run/nginx.conf:/etc/nginx/nginx.conf
    ports:
      - "80:80"
      - "443:443"

It's trying to mount local files instead of remote for some reason and fails with error:

ERROR: for nginx  Cannot start service nginx: OCI runtime create failed: container_linux.go:296: starting container process caused "process_linux.go:398: container init caused \"rootfs_linux.go:58: mounting \\\"/home/local-user/run/nginx.conf\\\" to rootfs \\\"/hdd/docker/overlay2/c869ef9f2c983d33245fe1b4360eb602d718786ba7d0245d36c40385f7afde65/merged\\\" at \\\"/hdd/docker/overlay2/c869ef9f2c983d33245fe1b4360eb602d718786ba7d0245d36c40385f7afde65/merged/etc/nginx/nginx.conf\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

Is it possible to mount remote resources via docker-compose similar to standard Docker CLI?

Answer 1

Docker contexts are kind of strange and, in my opinion, seem a bit half-baked. I'm aware that his is an old question, but after spending my afternoon tearing out my hair trying to get this to work for myself, I just want to put this info out here for everyone.

Interpolation is always local

All that docker compose (and docker stack, etc.) is is just a fancy frontend to the underlying backend API (the docker.sock protocol). This allows docker-compose to have a bunch of fancy features that docker engine itself doesn't have. One of those is the ability to use relative paths or expand variables like ${HOME} as you do in your example.

When you use docker-compose in a context, all that happens is that after docker-compose has done all its fancy magic locally on your machine, it then sends the low-level API calls over ssh to the remote engine. This means that whenver you use a fancy magic feature like putting $HOME or ./ in a path, docker compose expands those locally using your system's HOME, i.e. your own local home directory and then sends those requests to the remote engine, where the deployment subsequently fails because those paths don't exist there.

However, using absolute paths works just fine. If you use /home/user/run/nginx.conf instead of ${HOME}/run/nginx.conf, then, in my experience, everything should be fine.

Use named volumes instead

In your question, the file you were trying to mount was a config file, which means that you probably needed full manual control over it, and that's a perfectly valid reason to use a bind mount.

However, if, for example, you're running a database, you won't need to manually edit the database's data directory. As long as the database itself can interact with it, it's fine. In this case, you should use a regular volume instead. To do this, add a volumes: directive to the end of the file and list your needed volume names:

services:
  postgres:
    restart: always
    environment:
      POSTGRES_PASSWORD: example
    volumes:
      - psql-data:/var/lib/postgresql/data

volumes:
  psql-data:

If you're using docker swarm, please notice that volumes are not replicated by default, but there are third-party drivers out there and AFAIK the NFS driver option may also be of use, but I can't seem to find much documentation about it.

Use connection persistence

If you, as I did, have a strange issue where running any docker command via ssh takes an obscene amount of time, the solution seems to be setting up ssh to persist your connections. To do this, create a file ~/.ssh/config and add the following lines:

Host example.com
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%C
  ControlPersist 600

Replace example.com with the hostname or IP address of your remote host.

If you get any errors mentioning something like "can't bind to path" or "file not fount", you may need to create the ~/.ssh/sockets directory:

mkdir ~/.ssh/sockets

This is by far no conclusive list of the idiosyncrasies of remote docker contexts, but hopefully this can save some people some time.