Docker Node is Down after service restart

Question

It seems my server ran out of space and I was having some problems with some of the deployed docker stacks. Took me a while to figure it out, but eventually I did and removed a couple of containers and images to free some space.

I was able to run service docker restart and it worked. However, there are some problems:

• docker info says the swarm is "Pending"
• docker node ls shows the only node I have (Leader), it is available but it is down
• journalctl -f -u docker says `level=error msg="error removing task " error="incompatible value module=node/agent/worker node.id="

When running docker service ls, all services have 0/1 replicas.

This is the status when running docker node ls

"Status": {
    "State": "down",
    "Message": "heartbeat failure for node in \"unknown\" state",
    "Addr": "<ip and port>"
},
"ManagerStatus": {
    "Leader": true,
    "Reachability": "reachable",
    "Addr": "<ip and port>"
}

How can I get my services running again?

Answer 1

Sometimes when you restart or update your docker version the tasks.db file gets corrupted.

This is an open issue (#34827), some people have suggested a workaround to this issue moving the tasks.db file and testing if this fixes the issue then delete the tasks.db file. Docker automatically will create a new one for you.

You can find the tasks.db file in /var/lib/docker/swarm/worker/

I've faced the same issue recently and this workaround saved my day. I didn't lose any data related to my Stacks

Update October/19/2020

issue (#34827) is closed but the solution still the same, remove the tasks.db file

Answer 2

Option 1:

Wait. Sometimes it fixes itself.

Option 2 (May vary depending on OS):

systemctl stop docker
rm -Rf /var/lib/docker/swarm
systemctl start docker
docker swarm init