Why is Docker filling up /var/lib/docker/overlay2?

Question

Even though I have successfully (?) removed all Docker images and containers, the folder /var/lib/docker/overlay2 still is vast (152 GB). Why? How do I reduce the used disk size?

I have tried to rename the folder (in preparation for a possible removal of the folder) but that caused subsequent pull requests to fail.

To me it appears pretty unbelievable that Docker would need this amount of vast disk space just for later being able to pull an image again. Please enlighten me what is wrong or why it has to be this way.

List of commands run which should show what I have tried and the current status:

$ docker image prune --force
Total reclaimed space: 0B

$ docker system prune --force
Total reclaimed space: 0B

$ docker image prune -a --force
Total reclaimed space: 0B

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

$ docker container ls
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

$ du -h --max-depth=1 /var/lib/docker/overlay2 | sort -rh | head -25
152G    /var/lib/docker/overlay2
1.7G    /var/lib/docker/overlay2/ys1nmeu2aewhduj0dfykrnw8m
1.7G    /var/lib/docker/overlay2/ydqchhcaqokdokxzbh6htqa49
1.7G    /var/lib/docker/overlay2/xmffou5nk3zkrldlfllopxcab
1.7G    /var/lib/docker/overlay2/tjz58rjkote2c79veonb3s6qa
1.7G    /var/lib/docker/overlay2/rlnr04hlcudgoh6ujobtsu2ck
1.7G    /var/lib/docker/overlay2/r4ubwsmrorpr08k8o5rko9n98
1.7G    /var/lib/docker/overlay2/q8x21c9enjhpitt365smkmn4e
1.7G    /var/lib/docker/overlay2/ntr973uef37oweqlxr4kmaxps
1.7G    /var/lib/docker/overlay2/mcyasqzo2gry5dvjxoao1opws
1.7G    /var/lib/docker/overlay2/m2k4u58mob6e2db86qqu1e1f8
1.7G    /var/lib/docker/overlay2/lizesless03kch8j7kpk89rcf
1.7G    /var/lib/docker/overlay2/kmu7mjvsopr8o63onbsijb98j
1.7G    /var/lib/docker/overlay2/khgjwqry5drdy0jbwf47gr2lb
1.7G    /var/lib/docker/overlay2/gt70ur50vw3whq265vmpep7ay
1.7G    /var/lib/docker/overlay2/c3tm1fcuekmdreowrfcso7nd4
1.7G    /var/lib/docker/overlay2/7j93t64mt63arj6sewyyejwyo
1.7G    /var/lib/docker/overlay2/3ftxvvg2xg02xuwcb3ut3dq89
1.7G    /var/lib/docker/overlay2/0m3o3lw6b1ggs8m6z4uv6ueqf
1.4G    /var/lib/docker/overlay2/r82rfxme096cq5pg1xz1z5arg
1.4G    /var/lib/docker/overlay2/qric73hv1z3nx4k0zop3fvcm6
1.4G    /var/lib/docker/overlay2/oyb0a5ab5h642y30s6hawj4r9
1.4G    /var/lib/docker/overlay2/oqf9ltfoy36evnkuo8ga2uepl
1.4G    /var/lib/docker/overlay2/ntuwvljxxzqs2oxhgg3enyo7x
1.4G    /var/lib/docker/overlay2/l0oi2lxdrtg42hk2rznknqk0r

$ ls -l /var/lib/docker/overlay2
total 136
drwx------  4 root root     72 Nov 20 13:03 00ep8i7v5bdmhqsxdoikslr19
drwx------  4 root root     72 Feb 28 09:47 026x5e2xns6ui2acym19qfvl7
drwx------  4 root root     72 Apr  2 19:20 032y8d31damevtfymq6yzkyi4
drwx------  4 root root     72 Apr 23 13:42 03wwbyd4uge9u0auk94wwdlig
drwx------  4 root root     72 Jan 15 12:46 04cy91a19owwqu9hyw6vruhzo
drwx------  4 root root     72 Apr  2 14:44 051625a0f856b63ed67a3bc9c19f09fb1c90303b9536791dc88717cb7379ceeb
drwx------  4 root root     72 Dec  3 19:56 059fk19uw70p6fqzei6wnj8s2
drwx------  4 root root     72 Apr 21 15:03 059mddrhqegqhxv1ockejw9gs
drwx------  4 root root     72 Nov 28 11:26 069dwkz92m8fao6whxnj4x9vp
drwx------  4 root root     72 Feb 28 09:47 06h7qo5f70oyzaqgn1elbx5u8
drwx------  4 root root     72 Dec 18 13:27 0756fd640036fa92499cfdcf4bcc3081d9ec16c25eebe5964d5e12d22beb9991
drwx------  4 root root     72 Apr 20 11:32 09rk4gm6x2mcquc5cz0yvbawq
drwx------  4 root root     72 Apr  2 19:55 09scfio3qvtewzgc5bdwgw4f6
drwx------  4 root root     72 May  4 14:00 0ac2a09aa4a038981d37730e56dece4a3d28e80f261b68587c072b4012dc044a
drwx------  4 root root     72 Feb 25 14:19 0c399f5c349ec61ac175525c52533b069a52028354c1055894466e9b5430fbc3
drwx------  4 root root     72 May  4 14:00 0cac39b1382986a2d9a6690985792c04d03192337ea37ee06cb74f2f457b7bb7
drwx------  4 root root     72 Mar  5 08:41 0czco1xx3148slgwf8imdrk33
drwx------  4 root root     72 Apr 21 08:30 0gb2iqev9e7kr587l09u19eff
drwx------  4 root root     72 Feb 20 18:03 0gknqh4pyg46uzi6asskbf8xk
drwx------  4 root root     72 Jan  8 11:43 0gugiou3wqu53os4dageh77ty
drwx------  4 root root     72 Jan  7 11:31 0i8fd5jet6ieajyl2uo1xj2ai
.
.
.

$ docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:27:04 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:25:42 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Answer 1

You might have switched storage drivers somewhere along the way, so maybe docker is just cleaning out those drivers but leaving overlay2 as is (I still can't understand why would pulling images would fail).

Let's try this, run docker info and check what is your storage driver:

$ docker info

Containers: 0
Images: 0
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
<output truncated>

If it is not overlay2 (as appears above) try switching to it, and then prune docker images again and check if that cleaned up that folder.

Another possible solution is mentioned in this thread, people are commenting that clearing logs solves this problem, so try the following:

1. Remove all log files:

   find /var/lib/docker/containers/ -type f -name "*.log" -delete
   

2. Restart docker daemon (or entire machine):

   sudo systemctl restart docker
   

   or

   docker-compose down && docker-compose up -d
   

   or

   shutdown -r now
   

Answer 2

Thanks for your input and suggestions!

I believe that I am still using overlay2 as storage driver:

$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.8
 Storage Driver: overlay2
<output truncated>

I also cleared the logs and restarted the daemon and actually also the entire machine. The problem however remained.

In the end I solved it by stoping the deamon, removing the entire docker folder and restarting the deamon, as suggested above.

df -h
sudo systemctl stop docker
sudo mv /var/lib/docker /var/lib/docker_old
sudo systemctl start docker
sudo rm -rf /var/lib/docker_old
df -h

I fear however that this will not be a permanent solutions and that the problem will come back, but this will hopefully last another year. :)