Docker Stack tasks Rejected because of Permission denied error

Question

I tried to deploy a stack (contains one service) with docker stack deploy command and run 2 instance from my application on swarm! here is my docker-compose file:

version: "3"
services:
  server:
    image: makbn/thumbnailer-server:v1
    deploy:
      replicas: 2
      update_config:
        parallelism: 2
      resources:
        limits:
          cpus: "0.1"
          memory: 1024M
      restart_policy:
        condition: on-failure
    ports:
      - 8080:8080
    networks: 
      - server_network
networks: 
  server_network:

After running sudo docker stack deploy -c ./docker-compose.yml ts command, My stack successfully deployed to swarm, But here is my docker stack ps ts output:

ID                  NAME                IMAGE                         NODE                DESIRED STATE       CURRENT STATE             ERROR                              PORTS
8zmjp6wt47ki        ts_server.1         makbn/thumbnailer-server:v1   fanap9-lp           Ready               Rejected 2 seconds ago    "mkdir /var/lib/docker: permis…"   
88vizpnabbi8         \_ ts_server.1     makbn/thumbnailer-server:v1   fanap9-lp           Shutdown            Rejected 7 seconds ago    "mkdir /var/lib/docker: permis…"   
609hl7hy7tnu         \_ ts_server.1     makbn/thumbnailer-server:v1   fanap9-lp           Shutdown            Rejected 12 seconds ago   "mkdir /var/lib/docker: permis…"   
gif8fvbgdhpy         \_ ts_server.1     makbn/thumbnailer-server:v1   fanap9-lp           Shutdown            Rejected 17 seconds ago   "mkdir /var/lib/docker: permis…"   
x60x9c6albe5         \_ ts_server.1     makbn/thumbnailer-server:v1   fanap9-lp           Shutdown            Rejected 22 seconds ago   "mkdir /var/lib/docker: permis…"   
ugvbw1gpdp8e        ts_server.2         makbn/thumbnailer-server:v1   fanap9-lp           Ready               Rejected 2 seconds ago    "mkdir /var/lib/docker: permis…"   
p34x1kp55ch8         \_ ts_server.2     makbn/thumbnailer-server:v1   fanap9-lp           Shutdown            Rejected 7 seconds ago    "mkdir /var/lib/docker: permis…"   
3exxflukrl4y         \_ ts_server.2     makbn/thumbnailer-server:v1   fanap9-lp           Shutdown            Rejected 12 seconds ago   "mkdir /var/lib/docker: permis…"   
ud83xfj5nefj         \_ ts_server.2     makbn/thumbnailer-server:v1   fanap9-lp           Shutdown            Rejected 17 seconds ago   "mkdir /var/lib/docker: permis…"   
ws35na8up793         \_ ts_server.2     makbn/thumbnailer-server:v1   fanap9-lp           Shutdown            Rejected 22 seconds ago   "mkdir /var/lib/docker: permis…"   

And the full error message is mkdir /var/lib/docker: permission denied

And the other problem is when i tried to create stack with docker service create command services deployed and running successfully but i cant access to the service from my localhost! i have 2 questions:

1. what is the problem with deploy stack from compose file and why i get mkdir /var/lib/docker: permission denied error?
2. how to configure my service networke k to accessible from host?

Answer 1

(Note: only relevant to users who may have installed docker via snap)

I experienced this issue because I had snap installed docker, which meant that docker-compose was confused about my docker installation. The /var/lib/docker was giving file permission errors because my system was actually using docker at /snap/bin/docker

To check whether this is the case for you, you can run which docker. If it returns /usr/bin/docker, you've done the normal app install. If it returns something at the path /snap/ that means you've installed from the snap docker package (which is no longer maintained. Ref: https://github.com/docker/docker-snap)

To fix the issue, I first uninstalled the snap docker install using snap remove docker and then installed it through the docker setup instructions at https://docs.docker.com/install/linux/docker-ce/ubuntu/#docker-ee-customers. Once I did that, I could deploy stacks without issue.

I can't remember where the idea came into my head to snap install docker instead, but installing by setting up repositories and then running sudo apt install docker.io seems to be the way to go.