Docker Machine on Mac: Cannot see mounted Volumes on docker host/docker-machine? Where are volumes physically stored?

Question

Am on a Macbook Pro laptop and running docker-machine (0.5.0) and docker-compose (1.5.0) to get my containers going.

This means I'm using docker-machine to create my virtualbox boot2docker driven HOST machines, which will run my docker daemon and host all my containers.

I think I'm missing something critical with the concept of HOSTS and VOLUME, as they refer to Docker and the documentation.

This is my docker-compose.yml file (web simply builds the php:5.6-apache image):

web:
  restart: "always"
  build: ./docker-containers/web
  ports:
    - "8080:80"
  volumes:
    - ./src:/var/www/html
  links:
    - mysql:mysql

mysql:
  restart: "always"
  image: mysql:5.7
  volumes_from:
    - data
  ports:
    - "3306:3306"
  environment:
    - MYSQL_ROOT_PASSWORD=XXX

data:
  restart: "no"
  image: mysql:5.7
  volumes:
    - /var/lib/mysql
  command: "true" 

Docker Compose file documention for volumes is here: http://docs.docker.com/compose/compose-file/

It states for volumes - Mount paths as volumes, optionally specifying a path on the host machine (HOST:CONTAINER), or an access mode (HOST:CONTAINER:ro).

HOST in this case refers to my VM created by docker-machine, correct? Or my local macbook file system? Mounting a path on my VM to a container?

Under web I declare:

volumes:
  - ./src:/var/www/html

and this is mapping my local macbook file system ./src folder on my macbook pro to my web container. If my understanding is correct though, shouldn't it be mapping the ./src folder on my VM to /var/www/html within the web container?! In theory I think I should be required to COPY my local mac file system folder ./src to my VM first, and then I do this volume declaration. It seems docker-compose is magically doing it all at once though? confused

Lastly, we can see that I'm creating a data-only container to persist my mysql data. I've declared:

volumes:
   - /var/lib/mysql

Shouldn't this create a /var/lib/mysql folder on my HOST boot2docker VM and I could then navigate to this folder on the VM, yes/no? When I use docker-machine to ssh into my machine, and then navigate to /var/lib, there is NO mysql folder at all?! Why is it not being created? Is there something wrong with my configuration? :/

Thanks in advance! Any explanations as to what I'm doing wrong here would be greatly appreciated!

Answer 1

Ok there's a couple of points that need to be addressed here.

Lets start with what a docker volume is(Try to not think about your macbook or the vagrant machine at this point. Just be mindful of the fact that the dockers use a different filesystem, where ever it may reside at this point ): Maybe imagine it like this, in and of itself every volume in Docker is just a part of the internal file system docker uses. The containers can use theses volumes, like they were "small harddrives" that can be mounted by them and also shared between them (or mounted by two of them at the same time, like mounting a super fast version of some ftp server to two clients or whatever :P ).

In principal you can declare these volumes ( still not thinking about your computer/vagrant itself, just the dockers ;) ) via the Dockerfile's VOLUME instruction. Standard example, run one webserver container like so:

FROM: nginx
VOLUME /www

Now everything that goes into /www can in theory be mounted and unmounted from a container and also mounted to multiple containers. Now Nginx alone is boring, so we want to have php run over the files that nginx stores to produce some more fun content. => We need to mount that volume into some php-fpm container. Ergo in our compose file we'd do this

web:
  image: nginx
php:
  image: php-fpm
  volumes_from:
    - web

=> voila! every folder declared by a VOLUME directive in the nginx/web container will be visible in the php one. Important point to note here, whatever is in nginx's /www, will override whatever php has in /www. If you put the :ro, php can't even write to that folder :)

Now moving close to your issue, there's a second way to declare volumes, that does not require them being declared in the Dockerfile. This can be done by mounting volumes from the host (in this case your vagrant/boo2docker thingy). Let's discuss this as though we're running on a native Linux first.

If you were to put something like:

volumes:
 - /home/myuser/folder:/folder

in your docker-compose.yml, then this will mean that /home/myuser/folder will now be mounted into the docker. It will override whatever the docker has in /folder and just like the /www also be accessible from the thing that declared it. Now the Linux machine the docker daemon is running on.

So much for the theory :), in fact you probably just need the following advice to get your stuff going :):

The way boot2docker/docker-machine/kitematic and all these things deal with the issue is simply, that they first of all just mount a volume in the vagrant machine to the docker containers, and them simply also mount this thing into your Mac file system, hoping it will all work out :P

Now for the practical problem all of us using this (or just trying to help their coworkers into the world of sweet sweet Docker :P) on Mac are facing is permissions. I mean think about it ( root or some other user handles files in the container,the user vagrant might handle files in the vagrant host and then your Mac user "skalfyfan" handles those files in Mac. They all have different user id's and whatnot => many problems ensue with that, and somewhat depending on what you're actually running in Docker. Mysql and Apache are especially painful, because they do not run as root within the container. This means, they often have trouble writing to the Mac file system.

Before trying the second approach below, simply try putting your container volumes under you Mac home directory. This will resolve issues with MySQL in most cases as I have found over time. Btw: No need to declare full paths to volumes ./folder is fine and read relative to the place your docker-compose.yml resides!

Just put the compose-yml in your Mac users folder, that's all that matters. No chmod 777 -R :P will help you here, it just needs to be under your home folder :)

Still some apps ( Apache for example ) will still give you a hard time. The fact that the user id of whatever runs in the container differs from your Mac user id will make your life hell. In order to get around this, you need to adjust the user id as well as the user group in a way that doesn't conflict with your Mac's permissions. The group you want on a Mac is staff, a UID that works would be for example 1000. Hence you could put this at the end of your Dockerfile:

RUN usermod -u 1000 www-data
RUN usermod -G staff www-data

or

RUN usermod -u 1000 mysql
RUN usermod -G staff mysql

So as you have now learnt:

In theory I think I should be required to COPY my local mac file system folder ./src to my VM first, and then I do this volume declaration. It seems docker-compose is magically doing it all at once though?

Right on, it does that :)

Lastly, we can see that I'm creating a data-only container to persist my mysql data. I've declared: volumes: - /var/lib/mysql

This one you got wrong :) As explained, if you don't give a host folder, then Docker will persist this path. But only for this container and all will stay within the docker file system. Nothing is written to the host at all! This will always only happen if you give a host folder before the container folder!

Hope this helped :)