Error using mount command within Dockerfile

Question

The following Dockerfile throws an error when building it. Is there a way to be able to use mount with overlay while building the container?

Dockerfile

FROM ubuntu:15.10

RUN mkdir /var/data \
 && mkdir /var/data/delta \
 && mkdir /var/data/delta/changes \
 && mkdir /var/data/delta/workdir \
 && mkdir /var/data/merged \
 && mkdir /var/data/lower

RUN mount -t overlay overlay -o lowerdir=/var/data/lower,upperdir=/var/data/delta/changes,workdir=/var/data/delta/workdir /var/data/merged

Error

...

Step 2 : RUN mount -t overlay overlay -o lowerdir=/var/data/lower,upperdir=/var/data/delta/changes,workdir=/var/data/delta/workdir /var/data/merged
 ---> Running in 37434cc88e15
mount: overlay is write-protected, mounting read-only
mount: cannot mount overlay read-only
Removing intermediate container 37434cc88e15
The command '/bin/sh -c mount -t overlay overlay -o lowerdir=/var/data/lower,upperdir=/var/data/delta/changes,workdir=/var/data/delta/workdir /var/data/merged' returned a non-zero code: 32

Note

I tried running the below mount command on within a ubuntu:15.10 container and it gives the same error. If the container is started with --privileged the command works.

Answer 1

Is there a way to be able to use mount with overlay while building the container?

Short answer: No, and there won't be any time soon.

It seems the consensus is that any kind of privileged operation during build breaks the image portability contract, as it could potentially modify the host system. If a different system were to then pull and run such an image instead of building it from source, the host would be in an invalid state, from the perspective of the resultant container.

Remember, docker build works by completing each step/layer in the Dockerfile using (loosely) these actions:

1. Run the image of the last step/layer as a new container
2. Complete the operations for the current step/layer within the container
3. Commit the container (incl. new state) to a new image
4. Repeat for any further steps

Therefore, it's clearly possible for a privileged build operation to break out of the temporary container and touch the host in this scenario. No bueno.

So, what do?

---

Solution 1 (baked-in mount)

UPDATE — 2015-10-24: Fail. See Solution 2 below for a working implementation.

Note: YMMV depending upon Docker version, storage/graph driver, etc. Here's my docker info, for comparison:

Containers: 12
Images: 283
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.10-040110-generic
Operating System: Ubuntu 15.04
CPUs: 4
Total Memory: 7.598 GiB
Name: agthinkpad
ID: F6WH:LNV4:HH66:AHYY:OGNI:OTKN:UALY:RD52:R5L5:ZTGA:FYBT:SWA4
WARNING: No swap limit support

Well, I'm surprisingly finding it nigh impossible to bake the mount into an image via docker commit. The /proc filesystem, where a file representation of mount metadata is written by the kernel (more specifically /proc/self/mounts, from within a container), doesn't appear to be persisted by Docker at all. From what I can tell, anyway — /var/lib/docker/overlay/<container-root-lower>/root/proc is empty, and /var/lib/docker/overlay/<container-root-upper>/upper/proc doesn't exist.

I thought /proc could be manipulated via a volume, and did indeed find some 2+ year old references to bind mounting /proc:/proc to achieve things that probably nobody should even be attempting (like this, possibly? 😉), but it appears this doesn't work at all anymore. Attemping to bind mount /proc to a host directory or even just make a volume of it is now a fatal error, even with docker run --privileged:

Code: System error

Message: "/var/lib/docker/overlay/c091a331f26bed12f22f19d73b139ab0c5b9971ea24aabbfad9c6482805984c9/merged/proc"
 cannot be mounted because it is located inside "/proc"

Frames:
---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@37
---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@52
---
2: StartInitialization
Package: Error response from daemon: Cannot start container c091a331f26bed12f22f19d73b139ab0c5b9971ea24aabbfad9c6482805984c9: [8] System error: "/var/lib/docker/overlay/c091a331f26bed12f22f19d73b139ab0c5b9971ea24aabbfad9c6482805984c9/merged/proc" cannot be mounted because it is located inside "/proc"

In terms of a method that doesn't require running mount … inside containers spawned from the image via a startup/entrypoint script, I'm really not sure where to go from this point. Because /proc/self/mounts is managed by the kernel, let alone not writable, this may never be possible. Hopefully I'm overlooking something and someone can point me in the right direction.

If you *must* do this during image compilation you can roll your own builder script by doing something like the following: — Optionally, first create a `Dockerfile` and use the stock builder: FROM mybaseimg|scratch COPY ./a /tmp/a RUN foo … `docker build -t mynewimg .` — Write a shell script using a combination of: `CID=$(docker create mynewimg)`, [`docker cp …`](https://docs.docker.com/reference/commandline/cp/), `docker start $CID`, `docker exec|run … $CID …`, `docker stop $CID`, `docker commit $CID mynewimg`, etc. *(Edit: Preferably, use the [API](https://docs.docker.com/reference/api/remote_api_client_libraries/)!)* When you need to apply a privileged operation, you can escalate the `docker run` command, but `--privileged` is total overkill here. If you *think* you shouldn't need `--privileged` for something you almost certainly *don't*. For an OverlayFS/AuFS mount you will need to initialize `docker run` with `--cap-add=[SYS_ADMIN]`, and for Ubuntu (and possibly other) hosts with `AppArmor`, you also need `--security-opt=[apparmor:unconfined]` (or preferably an AppArmor profile which relaxes the necessary restrictions, instead of `unconfined`). I'm not sure about `SELinux`.

---

Solution 2 (runtime mounting)

host$uname -a

Linux agthinkpad 4.1.10-040110-generic #201510030837 SMP Sat Oct 3 12:38:41 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

host$mkdir /tmp/overlay-test && cd /tmp/overlay-test

./Dockerfile:

FROM debian:jessie

RUN apt-get update && apt-get install -y curl jq

WORKDIR /usr/local/sbin

# Locate and fetch the latest version of gosu
RUN ["/bin/bash", "-c", "curl -o ./gosu -sSL \"$( \
      curl -s https://api.github.com/repos/tianon/gosu/releases/latest \
      | jq --raw-output \
        '.assets[] | select(.name==\"gosu-'$(dpkg --print-architecture)'\") | .browser_download_url' \
    )\" && chmod +x ./gosu"]

COPY ./entrypoint.sh ./entrypoint
RUN chmod +x ./entrypoint

# UPPERDIR and WORKDIR **MUST BE ON THE SAME FILESYSTEM**, so
#  instead of creating a VOLUME for UPPERDIR we have to create a 
#  parent directory for both UPPERDIR and WORKDIR, and then make
#  it the VOLUME. 
RUN ["/bin/bash", "-c", "mkdir -p /var/overlay-test/{lower,upper/{data,work}} /mnt/overlay-test"]
VOLUME /var/overlay-test/upper

# Create a file named FOO in the lower/root branch
RUN touch /var/overlay-test/lower/FOO

ENTRYPOINT ["entrypoint"]

./entrypoint.sh:

#!/bin/bash
set -e

cd /var/overlay-test
mount -t overlay -o lowerdir=lower,upperdir=upper/data,workdir=upper/work overlay /mnt/overlay-test
chown -R "$DUID":"$DGID" ./
chown root: ./upper/work
chmod 0750 ./upper/work

cd /mnt/overlay-test
exec gosu "$DUID":"$DGID" $@

host$docker build -t overlay-test ./

Successfully built 582352b90f53

OK, let's test it!

Note: Under an Ubuntu 15.04 host I had trouble deleting (overlay whiteout) any files that exist in the lowerdir via the mounted directory inside the container. This bug appears to be the culprit: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1480411 — Edit: rm works all of a sudden and I can see the char file in upper/data, so I can only assume this is fixed and I received an updated package.

host$docker run -it --name=overlay-test --env="DUID=$(id -u)" --env="DGID=$(id -g)" --cap-add=SYS_ADMIN --security-opt=apparmor:unconfined overlay-test /bin/bash

overlay-test$id

uid=1000 gid=1000 groups=1000

overlay-test$mount | grep '/mnt/overlay-test'

overlay on /mnt/overlay-test type overlay (rw,relatime,lowerdir=lower,upperdir=upper/data,workdir=upper/work)

overlay-test$pwd

/mnt/overlay-test

overlay-test$ls -Al | sed '/^t/d'

-rw-r--r-- 1 1000 1000    0 Oct 24 03:54 FOO

overlay-test$touch BAR

overlay-test$ls -Al | sed '/^t/d'

-rw-r--r-- 1 1000 1000    0 Oct 24 04:21 BAR
-rw-r--r-- 1 1000 1000    0 Oct 24 03:54 FOO

overlay-test$ls -Al /var/overlay-test/{lower/,upper/*} | sed '/^t/d'

ls: cannot open directory /var/overlay-test/upper/work: Permission denied

/var/overlay-test/lower:
-rw-r--r-- 1 1000 1000 0 Oct 24 03:54 FOO

/var/overlay-test/upper/data:
-rw-r--r-- 1 1000 1000 0 Oct 24 04:21 BAR

So far so good…let's try importing the volume from another container:

overlay-test$exit

host$docker run --rm --user="$(id -u):$(id -g)" --volumes-from=overlay-test debian:jessie /bin/bash -c "ls -Al /var/overlay-test/upper/* | sed '/^t/d'"

ls: cannot open directory /var/overlay-test/upper/work: Permission denied

/var/overlay-test/upper/data:
-rw-r--r-- 1 1000 1000 0 Oct 24 05:32 BAR

Success! Note that you could also RUN echo the mount spec >> /etc/fstab in your Dockerfile and then mount -a inside the entrypoint script, but this method was quirky, in my experience. I'm not sure why, but as there isn't a functional difference between the two methods, I didn't bother to investigate any further.

Clean-up: host$docker rm -v overlay-test && docker rmi overlay-test

---

Documentation for container runtime security in Docker:

https://docs.docker.com/reference/run/#security-configuration https://docs.docker.com/reference/run/#runtime-privilege-linux-capabilities-and-lxc-configuration