docker-swarm and docker-compose how to dynamically add nodes and have them resolvable by the services

Question

I have been playing with docker-compose and have cobbled together a project from the docker hub website.

One thing that eludes me is how I can scale individual services up (by adding more instances) AND have existing instances somehow made aware of those new instances.

For example, the canonical docker-compose example comprises a cluster of:

• redis node
• python (flask) node
• haproxy load balancer

I create the cluster and everything works fine, however I attempt to add another node to the cluster:

$ docker-compose scale web=2
Creating and starting 2 ... done


$ docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                                          NAMES
e83f6ed94546        packetops/web:latest   "/bin/sh -c 'python /"   6 minutes ago       Up About a minute   80/tcp                                         swarm-slave/1_web_2
40e01a615a2f        tutum/haproxy          "python /haproxy/main"   7 minutes ago       Up About a minute   443/tcp, 1936/tcp, 172.16.186.165:80->80/tcp   swarm-slave/1_lb_1
f16357a28ac4        packetops/web:latest   "/bin/sh -c 'python /"   7 minutes ago       Up About a minute   80/tcp                                         swarm-slave/1_lb_1/1_web_1,swarm-slave/1_lb_1/web,swarm-slave/1_lb_1/web_1,swarm-slave/1_web_1
8dd59686e7be        redis                  "/entrypoint.sh redis"   8 minutes ago       Up About a minute   6379/tcp                                       swarm-slave/1_redis_1,swarm-slave/1_web_1/1_redis_1,swarm-slave/1_web_1/redis,swarm-slave/1_web_1/redis_1,swarm-slave/1_web_2/1_redis_1,swarm-slave/1_web_2/redis,swarm-slave/1_web_2/redis_1

That worked... But lets see what the haproxy node sees of the cluster (docker-machine modifies the '/etc/hosts' file)

# docker exec -i -t swarm-slave/1_lb_1 /bin/bash -c 'cat /etc/hosts'
172.17.0.4      40e01a615a2f
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix


ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.3      1_web_1 f16357a28ac4
172.17.0.3      web f16357a28ac4 1_web_1
172.17.0.3      web_1 f16357a28ac4 1_web_1

If I were to restart the entire cluster using docker-compose that node should have it's /etc/hosts populated but it now seems to have broken even further:

$ docker-compose up --force-recreate -d
Recreating 1_redis_1
Recreating 1_web_2

Recreating 1_web_1
Recreating 1_lb_1
ERROR: Unable to find a node fulfilling all dependencies: --link=1_web_1:1_web_1 --link=1_web_1:web --link=1_web_1:web_1 --link=1_web_2:1_web_2 --link=1_web_2:web --link=1_web_2:web_2

$ docker-compose up -d
1_redis_1 is up-to-date
1_web_1 is up-to-date
1_web_2 is up-to-date
Starting 40e01a615a_1_lb_1

$ docker exec -i -t swarm-slave/40e01a615a_1_lb_1  /bin/bash -c 'cat /etc/hosts'
172.17.0.4      40e01a615a2f
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

So in conclusion is there a smarter way to do this (resolution and discovery)? Is there another smarter way rather than just updating the hosts files ? What's the best practice here?

Answer 1

Docker just released a new Version with built in orchestration:

https://blog.docker.com/2016/06/docker-1-12-built-in-orchestration/

You can start a new Swarm Cluster with:

docker swarm init

And create Services:

docker service create –name frontend –replicas 5 -p 80:80/tcp nginx:latest

The created Services will be load balanced and you can scale it up and down:

docker service scale frontend=X