Docker container discovery by hostnames doesn't work

Question

I use the following docker compose file located at /tmp/compose.yml for docker stack deployment :

version: "3.6"

services:
  service-A:
    image: service-A
    networks:
      - net
    hostname: "service-A-{{.Task.Slot}}"
    deploy:
      replicas: 2

  service-B:
    image: service-B
    networks:
      - net
    hostname: "service-B-{{.Task.Slot}}"
    deploy:
      replicas: 2

networks:

  net:

Run commands:

docker network create -d overlay net
docker stack deploy -c /tmp/compose.yml my

My expectation was that service-A individual containers will be able to discover service-B containers by container pre defined hostnames: service-B-1, service-B-2. However, ping service-B-1 from within any service-A container fails.

Docs state:

hostname Sets the hostname by which the container knows itself. This is written into /etc/hostname, into /etc/hosts as the name of the container's host-facing IP address, and is the name that /bin/bash inside the container will display inside its prompt. But the hostname is not easy to see from outside the container. It will not appear in docker ps nor in the /etc/hosts file of any other container.

How to achieve docker swarm service container to service container communication by container pre defined hostnames?

The only way that works is adding entries like:

[service_B_container_1_virtual_ip] service-B-1
[service_B_container_2_virtual_ip] service-B-2

to /etc/hosts of service-A containers.

It's interesting also that if hostname is not set in docker-composes file, but generated by docker, ping [container_id] succeeds even across multi host docker swarm.

I expected the same behavior for pre defined hostnames.

Answer 1

I don't think there is any straightforward way to do this. And the more I think about it, the more it makes sense that there should not be a way to do what you are asking, by design.

One of the fundamental properties of a stack service is that it should be deployable in such a way that Docker is able to treat all of the replicas in that service interchangeably. If you need to connect to individual instances, it suggests that those containers should not be created as service replicas in the first place, as there is something that distinguishes them. A proper stack service should be able to treat every replica as an ephemeral entity that can be destroyed and rebuilt at any time, and the operation should be invisible from the outside.

What you can do from inside the Docker network is reference the service entity using the service name. So in your example, ping Service-A should work from any container in the stack, and so will ping Service-B.

The names of individual containers in a service are things like

my_stack_redis.1.y1pwzeg8239yb6n6ynefs78br
my_stack_webserver.1.d6cxx8nlg7db9m2poy9wfjdyd

If you absolutely must gain access to those names from within a container, the only way I can think of is to expose /var/run/docker.sock inside the container via bind mount when starting. Then, chmod 750 on the socket as part of the CMD / ENTRYPOINT script so a non-root user can read it. Now you should be able to install the Docker CLI and use docker network inspect to discover the names of actual service members, and jq to parse them out (you'll probably need to add both packages to the image). But at that point, the better option would probably be to separate those non-fungible containers into their own service groups.