docker - driver "devicemapper" failed to remove root filesystem after process in container killed

Question

I am using Docker version 17.06.0-ce on Redhat with devicemapper storage. I am launching a container running a long-running service. The master process inside the container sometimes dies for whatever reason. I get the following error message.

/bin/bash: line 1: 40 Killed python -u scripts/server.py start go

I would like the container to exit and to be restarted by docker. However docker never exits. If I do it manually I get the following error:

Error response from daemon: driver "devicemapper" failed to remove root filesystem.

After googling, I tried a bunch of things:

docker rm -f <container>
rm -f <pth to mount>
umount <pth to mount>

All result in device is busy. The only remedy right now is to reboot the host system which is obviously not a long-term solution.

Any ideas?

Answer 1

I had the same problem and the solution was a real surprise.

So here is the error om docker rm:

$ docker rm 08d51aad0e74
Error response from daemon: driver "devicemapper" failed to remove root filesystem for 08d51aad0e74060f54bba36268386fe991eff74570e7ee29b7c4d74047d809aa: remove /var/lib/docker/devicemapper/mnt/670cdbd30a3627ae4801044d32a423284b540c5057002dd010186c69b6cc7eea: device or resource busy

Then I did the following (basically go through all processes and look for docker in mountinfo):

$  grep docker /proc/*/mountinfo | grep 958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac
/proc/20416/mountinfo:629 574 253:15 / /var/lib/docker/devicemapper/mnt/958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac rw,relatime shared:288 - xfs /dev/mapper/docker-253:5-786536-958722d105f8586978361409c9d70aff17c0af3a1970cb3c2fb7908fe5a310ac rw,nouuid,attr2,inode64,logbsize=64k,sunit=128,swidth=128,noquota

This got be the PID of the offending process keeping it busy - 20416 (the item after /proc/)

So I did a ps -p and to my surprise find:

[devops@dp01app5030 SeGrid]$ ps -p 20416
  PID TTY          TIME CMD
20416 ?        00:00:19 ntpd

A true WTF moment. So I pair problem solved with Google and found this: Then found this https://github.com/docker/for-linux/issues/124

Turns out I had to restart ntp daemon and that fixed the issue!!!