How to use Docker swarm DNS/Service names in Nginx upstream

Question

I have a service running across 4 swarm nodes (ServiceA) and a Nginx service running across 4 nodes on the same Swarm. The Nginx service exposes/publishes ports 80 and 443. All the services are connected to the same user-defined overlay network and most importantly I can curl/ping the service name (ServiceA) from within the containers so all is working so far.

My question is how do I get Nginx upstream to work with the service names? I've read a lot and tried adding this to the nginx.conf resolver 127.0.0.11 ipv6=off; but it has not helped and the Nginx service will not start. Any ideas on how to get Nginx see the Docker network DNS names?

This is my nginx.conf

events { 
    worker_connections 4096; 
}

http {
    include /etc/nginx/conf/*.conf;
    include /etc/nginx/mime.types;
    proxy_intercept_errors off;
    proxy_send_timeout 120;
    proxy_read_timeout 300;

    upstream serviceA {
        ip_hash;
        server serviceA:8081;
    }

    server {
        listen 80 default_server;
        resolver 127.0.0.11 ipv6=off;
        keepalive_timeout  5 5;
        proxy_buffering    off;
        underscores_in_headers on;

        location ~ ^/serviceA(?<section>.*) {
            access_log /var/log/nginx/access.log nginx_proxy_upstream;
            proxy_pass http://serviceA/$section$is_args$query_string;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }

    server {
        listen 443 ssl;
        resolver 127.0.0.11 ipv6=off;
        keepalive_timeout  5 5;
        proxy_buffering    off;
        underscores_in_headers on;

        # allow large uploads
        client_max_body_size 10G;

        ssl_certificate /etc/nginx/ssl/myKey.crt;
        ssl_certificate_key /etc/nginx/ssl/myKey.key;

        location ~ ^/serviceA(?<section>.*) {
            access_log /var/log/nginx/access.log nginx_proxy_upstream;
            proxy_pass http://serviceA/$section$is_args$query_string;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

    }
}

Answer 1

Removing the resolver should work if the upstream containers have already been deployed (DNS entries created). However this means that you cannot start nginx unless the upstream containers are already running.

For a dynamic approach via resolver, you need to make docker engine host's DNS accessible from within the container (not via 127.0.0.11 ... which is the container itself UPDATE: on custom networks it is possible to query 127.0.0.0/8 addresses).

https://docs.docker.com/engine/userguide/networking/configure-dns/ :

Note: If you need access to a host’s localhost resolver, you must modify your DNS service on the host to listen on a non-localhost address that is reachable from within the container.

UPDATE: I managed to do it on a custom overlay network in docker swarm like this:

  location / {
    resolver 127.0.0.11 ipv6=off;
    set $upstream_addr <swarm_stack_name>:<port>;
    proxy_pass https://$upstream_addr;
    ...
  }

I did not get it to work with the upstream {} nginx directive ... this does not seem to handle dynamic resolves or I overlooked something.