Direct-LVM stops working after reboot

Question

In order to run Docker in production, I am following the steps in https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#configure-direct-lvm-mode-for-production

Everything seems to be working fine. But if I reboot for some reason, everything messes up. Even if I remove everything from /var/lib/docker and I run lvremove, vgremove and pvremove it still says Error starting daemon: error initializing graphdriver: devmapper: Unable to take ownership of thin-pool (docker-thinpool) that already has used data blocks and resists to start.

I know there has to be some documentation that shows how to make direct-lvm settings persistent against reboots. Something automatically restores the settings after reboot. I could not find any.

So how do I achieve persistence for my direct-lvm settings?

Answer 1

Fortunately, someone in Docker Community understood the problem and shared their solution with. The solution is: https://github.com/projectatomic/docker-storage-setup .

apt install -y thin-provisioning-tools
mkdir /usr/lib/docker-storage-setup
mkdir /etc/sysconfig
git clone https://github.com/projectatomic/docker-storage-setup.git     /opt/docker-storage-setup
cp /opt/docker-storage-setup/docker-storage-setup.sh /usr/bin/docker-storage-setup
cp /opt/docker-storage-setup/docker-storage-setup.service /lib/systemd/system/docker-storage-setup.service
cp /opt/docker-storage-setup/libdss.sh /usr/lib/docker-storage-setup
VG=docker DATA_SIZE=95%FREE STORAGE_DRIVER=devicemapper /opt/docker-storage-setup/docker-storage-setup.sh

systemctl enable docker-storage-setup
lvrename docker/thinpool docker/docker-pool

And the related section in systemd service file in /lib/systemd/system/docker.service needs to be updated as --storage-opt=dm.thinpooldev=/dev/mapper/docker-docker--pool

Answer 2

In a bug report Eric Paris says:

IF you are using device mapper (instead of loopback) /var/lib/docker contains metadata informing docker about the contents of the device mapper storage area. If you delete /var/lib/docker that metadata is lost. Docker is then able to detect that the thin pool has data but docker is unable to make use of that information. The only solution is to delete the thin pool and recreate it so that both the thin pool and the metadata in /var/lib/docker will be empty.

I ran into the same problem because of the documentation's wording you mentioned. There was the step rm -rf /var/lib/docker.bk and it was only then that I removed the original files, causing the failure.

With pvremove -ff /dev/sda2 (my lvm drive) and recreating, wiping the signatures in the lvm partitions, it worked for me.

I think with your settings it should already be persistent.

For me another error occured, after reboot lsblk did not show my lvm volumes, neither ls /dev/mapper. I am using Ubuntu and a commit message says that its default setup doesn't (fully?) support thin provisioning. After sudo apt-get install thin-provisioning-tools the command sudo vgchange -ay docker as well as reboots worked for me.