Why to chose role as worker for nodes in docker swarm clusters at all?

Question

Lets say we have a test setup of 10 nodes, 4 managers and 6 workers.

When the leader manager fails, the other 3 managers will chose another manager as leader.

When this leader as well fails, we only have 2 managers left out of 4. The other managers then say

Error response from daemon: rpc error: code = Unknown desc = The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online.

Because we have not more than half of the managers left, they will not be able to chose a new leader although 2 managers of the cluster are left.

My question is

1. the sense of this rule, because the cluster is without a leader and not manageable anymore as long as no additional managers are added to the cluster, although there are 2 managers available.
2. Why should I chose the role worker for nodes at all? What advantage are there to have nodes as workers? Managers also act as workers by default only with the disadvantage that they cannot take over when manager nodes fail.

Answer 1

1. Docker recommends to use a system with odd number of manager nodes. So your initial setup of 4 manager is as good as having 3 manager nodes. It is recommended that you start with 5 nodes, as you are loosing 2 nodes. Also, isn't there any serious issue to be addressed in the way you are using? (loosing so many nodes is not a good sign)

If the swarm loses the quorum of managers, the swarm cannot perform management tasks. If your swarm has multiple managers, always have more than two. To maintain quorum, a majority of managers must be available. An odd number of managers is recommended, because the next even number does not make the quorum easier to keep. For instance, whether you have 3 or 4 managers, you can still only lose 1 manager and maintain the quorum. If you have 5 or 6 managers, you can still only lose two.

2. Having a dedicated worker nodes makes sure that they won't participate in the Raft distributed state, make scheduling decisions, or serve the swarm mode HTTP API. So the complete compute power of these nodes are dedicated specifically to run the containers.

because manager nodes use the Raft consensus algorithm to replicate data in a consistent way, they are sensitive to resource starvation

The quotes are taken from the docker official documentation link