Docker run error: "Thin Pool has free data blocks which is less than minimum required"

Question

We are trying to run a docker in a way that used to work, but now we get a "Thin Pool lack of space" error:

docker run --privileged -d --net=host --name=fat-redis -v /fat/deploy:/fat/deploy -v /fat/fat-redis/var/log:/var/log -v /home:/home fat-local.indy.xiolab.myserv.com/fat-redis:latest /fat/deploy/docker/fat-redis/fat_start_docker_inner.sh
docker: Error response from daemon: devmapper: Thin Pool has 486 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior.
See 'docker run --help'.

What does this error mean? We tried 'docker rmi' and the advise from here, but all in vain.

Any ideas?

Thank you

Answer 1

Running with data/metadata over loopback devices was the default on older versions of docker. There are problems with this, and newer versions have changed this default. If docker was configured this way, then normal updates (e.g. through rpm/apt) don't change the configuration, which is why a full reinstall was required to fix.

Here's an article with instructions on how to configure older versions to not use loopback devices: http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/

Answer 2

You don't have to reinstall Docker. Rather, you can clean up all the containers, images, volumes, etc. under /var/lib/docker directory.

Those images could be pulled up from your Docker repositories again. (This is assuming you only use this Docker host for building Docker images.)

Answer 3

TL;DR

Sometimes you just need more space. Increase the data file with the truncate command.

Explanation: The reason that a reinstall or a purge of all your images works is that you have a "ramdisk" that docker uses as a space to build the images, but it's not purged after the image is running. If you are running several different images, you can fill up the scratch disk and the "newer" image doesn't have enough space to run in. The docker system prune command won't work because that space is legitimately consumed. You need to increase the size of the scratch file.

1. Make sure you have extra physical space on disk

   df

2. Figure out the size of your data file

   docker info |grep 'Data Space'

3. Find the location of your data file

   docker info |grep 'loop file'

4. Increase the size of your data file (+50G or whatever)

   sudo truncate -s 150G /var/lib/docker/devicemapper/devicemapper/data

5. Restart the machine. The guide talks about a bunch of commands to "cascade" the resize through the layer, but a restart did them automatically

   sudo reboot

References:

{all the SO posts that complained about the loopback driver being outdated} https://docs.docker.com/storage/storagedriver/device-mapper-driver/#use-operating-system-utilities

Answer 4

Use the following to cleanup unnecessary images.

docker image prune -a --force --filter "until=240h"

Refer to this document for more details: https://docs.docker.com/engine/reference/commandline/image_prune/