docker swarm - how to balance already running containers in a swarm cluster?

Question

I have docker swarm cluster with 2 nodes on AWS. I stopped the both instances and initially started swarm manager and then worker. Before stopped the instances i had a service running with 4 replicas distributed among manager and worker.
When i started swarm manager node first all replica containers started on manager itself and not moving to worker at all.
Please tell me how to do load balance?
Is swarm manager not responsible to do when worker started?

Answer 1

Swarm currently (18.03) does not move or replace containers when new nodes are started, if services are in the default "replicated mode". This is by design. If I were to add a new node, I don't necessarily want a bunch of other containers stopped, and new ones created on my new node. Swarm only stops containers to "move" replicas when it has to (in replicated mode).

docker service update --force <servicename> will rebalance a service across all nodes that match its requirements and constraints.

Further advice: Like other container orchestrators, you need to give capacity on your nodes in order to handle the workloads of any service replicas that move during outages. You're spare capacity should match the level of redundancy you plan to support. If you want to handle capacity for 2 nodes failing at once, for instance, you'd need a minimum percentage of resources on all nodes for those workloads to shift to other nodes.

Answer 2

Swarm doesn't do auto-balancing once containers are created. You can scale up/down once all your workers are up and it will distribute containers per your config requirements/roles/etc.

see: https://github.com/moby/moby/issues/24103

There are problems with new nodes getting "mugged" as they are added. We also avoid pre-emption of healthy tasks. Rebalancing is done over time, rather than killing working processes. Pre-emption is being considered for the future.

As a workaround, scaling a service up and down should rebalance the tasks. You can also trigger a rolling update, as that will reschedule new tasks.