Docker : Overlay2 size too big

Question

I'm running docker's environment with two containers.I noted the overlay2 folder size is too big. When the docker is down (docker-compose down) the overlay2 folder is 2.3GB size. When the containers are running, the overlay2 folder increase to 4.0GB and it's increasing by the time. Is it normal?

The command du -shc /var/lib/docker/* with the containers stoped:

76K     /var/lib/docker/buildkit
268K    /var/lib/docker/containers
3.7M    /var/lib/docker/image
64K     /var/lib/docker/network
2.3G    /var/lib/docker/overlay2
0       /var/lib/docker/plugins
0       /var/lib/docker/runtimes
0       /var/lib/docker/swarm
0       /var/lib/docker/tmp
0       /var/lib/docker/tmp-old
0       /var/lib/docker/trust
236M    /var/lib/docker/volumes
2.5G    total

The command du -shc /var/lib/docker/* with the containers running:

76K     /var/lib/docker/buildkit
448K    /var/lib/docker/containers
4.9M    /var/lib/docker/image
64K     /var/lib/docker/network
4.0G    /var/lib/docker/overlay2
0       /var/lib/docker/plugins
0       /var/lib/docker/runtimes
0       /var/lib/docker/swarm
0       /var/lib/docker/tmp
0       /var/lib/docker/tmp-old
0       /var/lib/docker/trust
235M    /var/lib/docker/volumes
4.3G    total

EDIT

The command docker system df

TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          4         3         1.472GB   750.4MB (50%)
Containers      13        2         106.9MB   89.31MB (83%)
Local Volumes   62        1         1.884GB   1.817GB (96%)
Build Cache     0         0         0B        0B

Answer 1

First, run docker system df. It displays information regarding the amount of disk space used by the docker daemon. You can see here.

Here is an example. You have 5 images, 2 actives and the local volume with one in inactive. So you can reclaim a free space. For each type you can reclaim a free space.

1. image : you can remove images unused with the command line docker image rmi <image name>. More info here
2. containers: you can remove an inactive containers with the command line docker container rm <container name>. More info here
3. local volumes: you can reclaim a free space using the command line docker volume prune. More info here

After all these steps, you can check the Overlay2 consume. If it's ok or not: du -shc /var/lib/docker/* .

The garbage collection is a misterios in all tecnologias. You think it's work like this, but it works in other way around. You can learn more here.

ATTENTION: Be careful using the docker system prune. Learn more here.

Answer 2

This is an indication the container's read-write layer has files being written to it by the app running inside the container. You can see what files are being created/modified with:

docker container diff ${container_name_or_id}

for your specific containers. If your apps are writing logs to the container, I'd recommend against that (best practice is to write to stdout and then limit the size of those logs using this answer).

If the apps change permissions or ownership on a file, or make a small change to an otherwise large file, that will trigger a copy-on-write of the entire file, which can quickly increase the size on disk.

Typically you should avoid modifying files from your image. They should be static, where you could hopefully change the entire container filesystem to read-only. Configs would be injected externally (with a volume mount or environment variable), and data would be stored in either a volume or external database.