Docker error - cannot remove a container

Question

I haven't upgraded Docker recently. I used to be able to remove containers without error. All the time now I get:

docker rm -f 05344fa394a4

Error response from daemon: driver "overlay" failed to remove root filesystem for 
05344fa394a43e5080deb1a43fdeab3a6f141038069f1a49209e29ec8f06a20c: 
remove /var/lib/docker/overlay/c9eb21a91ae1a37a121855a1ef65a89593994dc036affa9ff295b59b4eca1af5/merged: 
device or resource busy

I basically have to reboot (systemctl restart docker does NOT fix) in order to successfully remove and continue. Anyone have any insight as to what might be wrong?

I am running on Centos 7.3 as root with Docker 17.06. Everything in /var/lib/docker and /var/lib/docker/overlay is owned by root:root is protection 700. [/var/lib/docker/network is actually 750]

Answer 1

This is a known issue being worked on by docker. I'd recommend upgrading to a newer kernel (the 7.4 centos equivalent is a large improvement) and migrating to overlay2. Once the 18.03 release comes out, I'd recommend upgrading since some of the fixes from the docker side will be included there.

When deleting containers causes this issue, you can try the force option (docker rm -f), which will often still report the error but ultimately remove the container after a few seconds. You'll likely have some leftover pieces that would be best cleaned up by a reboot, but it may help workaround any other issues the filesystem locking problem is causing.

One other thing I've seen improve docker on RH and CentOS systems is to make a file with:

$ cat /etc/systemd/system/docker.service.d/override.conf
[Service]
MountFlags=slave

This should now be the default in current releases of docker, but you may have copied your unit file from an older version that didn't have it. To apply that file, you'll need to systemctl daemon-reload && systemctl restart docker as root.

Answer 2

According to the Plesk Site, a solution would be

1. Stop the container
2. Reboot the server
3. Remove the container

Or, if you are running proxy rules, you have to remove them at first:

1. Remove proxy rule, stop container, remove container -> error
2. Stop container, remove proxy rule, remove container -> works

In case of a), if you already removed the proxy rule, creating another container and adding a new proxy rule (same domain/subdomain) towards the new one will allow you to remove the older one immediately.

This seems somehow to be linked to modifying nginx config file after the container is stopped.