docker swarm services stuck in preparing

Question

I have a swarm stack deployed and I removed couple services from the stack and tried to deploy them again. these services are showing with desired state remove and current state preparing .. also their name got changed from the custom service name to a random docker name. swarm also trying to start these services which are also stuck in preparing. I ran docker system prune on all nodes and them removed the stack. all the services in the stack are not existent anymore except for the random ones. now I cant delete them and they still in preparing state. the services are not running anywhere in the swarm but I want to know if there is a way to remove them.

Answer 1

I had the same problem. Later I found that the current state, 'Preparing' indicates that docker is trying to pull images from docker hub. But there is no clear indicator in docker service logs <serviceName> available in the docker-compose-version above '3.1'.

But it sometimes imposes the latency due to n\w bandwidth or other docker internal reasons.

Hope it helps! I will update the answer if I find more relevant information.

P.S. I identified that docker stack deploy -c <your-compose-file> <appGroupName> is not stuck when switching the command to docker-compose up. For me, it took 20+ minutes to download my image for some reasons.

So, it proves that there is no open issues with docker stack deploy,

Adding reference from Christian to club and complete this answer.

Use docker-machine ssh to connect to a particular machine:

docker-machine ssh <nameOfNode/Machine>

Your prompt will change. You are now inside another machine. Inside this other machine do this:

tail -f /var/log/docker.log

You'll see the "daemon" log for that machine. There you'll see if that particular daemon is doing the "pull" or what's is doing as part of the service preparation. In my case, I found something like this:

time="2016-09-05T19:04:07.881790998Z" level=debug msg="pull progress map[progress:[===========================================>       ] 112.4 MB/130.2 MB status:Downloading

Which made me realise that it was just downloading some images from my docker account.