Docker: Monitor disk writes to the container, i.e. by the overlay storage driver

Question

I would like to monitor data written "inside" a Docker container, meaning data written to the backing filesystem by the overlay storage driver. Not data written to volumes, tmpfs or bind mounts. Typical monitoring tools, such as docker stats seem to report the total amount of data written.

BLOCK I/O The amount of data the container has read to and written from [sic] block devices on the host

Source: docker stats

The idea is to keep containers as read-only as possible, by finding "write-heavy" files / folders and moving them to volumes or bind mounts. So an ideal solution would not (only) show the data currently written, but the total amount of data written since the container was started, ideally breaking it down to single files.

At the moment I'm simply using find -type f -mtime x from the container shell, where x is a smaller than the image age, but there must be a better solution for this.

I'm using: Server Version: 18.06.1-ce, Storage Driver: overlay2, Backing Filesystem: extfs

Answer 1

Actually the docker storage driver itself provides the answer already.

Taking the overlay2 storage driver, which is the default driver on most distributions, as an example, we see that the container layer, where all data written to the container is stored, is kept in a separate folder:

Source: How the overlay driver works

Total amount of data written to the container layer

For a complete overview of what has been written to the container, we only have to take a look at the upperdir, which is called diff on the backing (host) file system. The path of the diff folder can be found with

docker container inspect <container_name> --format='{{.GraphDriver.Data.UpperDir}}'   # or
docker container inspect <container_name> | grep UpperDir

With default settings, this path points to /var/lib/docker/overlay2/. Note that access to the "inner workings" of docker requires root access on the host, and it's a good idea not to do any writes to these folders.

Now that we have the folder on the backing file system, we can simply du in much detail as we want. As a test example, I've used an alpine image that runs a script, which writes a 10 MB dummy file every 10 seconds.

root@testbox:/var/lib/docker/overlay2/83a825d...# du -h -d 1
8.0K    ./work
216M    ./diff
216M    .
root@testbox:/var/lib/docker/overlay2/83a825d...# ll diff/tmp
total 220164
drwxrwxrwt 2 root root    4096 Okt 21 22:57 ./
drwxr-xr-x 3 root root    4096 Okt 21 22:53 ../
-rw-r--r-- 1 root root 9266613 Okt 21 22:53 dummy0.tar.gz
-rw-r--r-- 1 root root 9266613 Okt 21 22:55 dummy10.tar.gz
-rw-r--r-- 1 root root 9266613 Okt 21 22:55 dummy11.tar.gz
[...]

Hence, seeing all the files and folders written to the container is as easy as with any other directory.