Docker overlay2 eating Disk Space

Question

Below is the file system in overlay2 eating disk space, on Ubuntu Linux 18.04 LTS

Disk space of server 125GB

overlay         124G  6.0G  113G   6% /var/lib/docker/overlay2/9ac0eb938cd2a50bb87e8ed13605d3f09214fdd9c8967f18dfc3f9432701fea7/merged
overlay         124G  6.0G  113G   6% /var/lib/docker/overlay2/397b099799212060ee7a4718660aa13aba8aa1fbb92f4d88d86fbad94e572847/merged
shm              64M     0   64M   0% /var/lib/docker/containers/7ffb129016d187a61a31c33f9e468b98d0ac7ab1771b87631f6caade5b84adc6/mounts/shm
overlay         124G  6.0G  113G   6% /var/lib/docker/overlay2/df7c4acee73f7aa2536d2a8929a48241bc8e92a5f7b9cb63ab70cea731b52cec/merged

Answer 1

Follow the Steps if your Server is Linux Ubuntu 18.04 LTS (should work for others too)

Docker info for Overlay2

Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true

if you got the following lines when you enter df -h --total

19M /var/lib/docker/overlay2/00d82017328c49c661c78ce14550c4073c50a550fe5004911bd3488b085aea76/diff
5.9M /var/lib/docker/overlay2/00e3e4fa0cbff7c242c38cfc9501ef1a523158d69b50779e08a773e7e22a01f1/diff
44M /var/lib/docker/overlay2/0e8e7e893b2c8aa17b4875d421670e058e4d97de066c970bbeab6cba566a44ba/diff
28K /var/lib/docker/overlay2/12a4c4e4877d35e9db657e4acff32e513042cb44119cca5c43fc19ad81c3915f/diff
............
............

then do the changes as follows:

First stop docker : sudo systemctl stop docker

Next: got to path /etc/docker

Check file daemon.json if not found

cat > daemon.json

and enter the following inside:

{
  "storage-driver": "aufs"
}

and close

Finally restart docker : sudo systemctl start docker

Check if the changes have been made:

Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 0
Dirperm1 Supported: true

Changing the file system can help you to resolve this issue.

Please if check your docker version supports aufs here:

Please do check the Linux distribution and what storage drivers supported here :

Answer 2

In case someone else runs into this, here's what's happening:

Your container may be writing data (logs, deployables, downloads...) to its local filesystem, and overlay2 will create a diff on each append/create/delete, so the container's filesystem will keep growing until it fills all available space on the host.

There are a few workarounds that won't require changing the storage driver:

1. first of all, make sure the data saved by the container may be discarded (you probably don't want to delete your database or anything similar)

2. periodically stop the container, prune the system docker system prune and restart the container

3. make sure the container doesn't write to its local filesystem, but if you can't:

4. replace any directories the container writes to with volumes or mounts.