nginx does not automatically pick up dns changes in swarm

Question

I'm running nginx via lets-nginx in the default nginx configuration (as per the lets-nginx project) in a docker swarm:

services:

  ssl:
    image: smashwilson/lets-nginx
    networks:
      - backend
    environment:
      - [email protected]
      - DOMAIN=api.finestructure.co
      - UPSTREAM=api:5000
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - letsencrypt:/etc/letsencrypt
      - dhparam_cache:/cache

  api:
    image: registry.gitlab.com/project_name/image_name:0.1
    networks:
      - backend
    environment:
      - APP_SETTINGS=/api.cfg
    configs:
      - source: api_config
        target: /api.cfg
    command:
      - run
      - -w
      - tornado
      - -p
      - "5000"

api is a flask app that runs on port 5000 on the swarm overlay network backend.

When services are initially started up everything works fine. However, whenever I update the api in a way that makes the api container move between nodes in the three node swarm, nginx fails to route traffic to the new container.

I can see in the nginx logs that it sticks to the old internal ip, for instance 10.0.0.2, when the new container is now on 10.0.0.4.

In order to make nginx 'see' the new IP I need to either restart the nginx container or docker exec into it and kill -HUP the nginx process.

Is there a better and automatic way to make the nginx container refresh its name resolution?

Answer 1

Thanks to @Moema's pointer I've come up with a solution to this. The default configuration of lets-nginx needs to be tweaked as follows to make nginx pick up IP changes:

  resolver 127.0.0.11 ipv6=off valid=10s;
  set $upstream http://${UPSTREAM};
  proxy_pass $upstream;

This uses docker swarm's resolver with a TTL and sets a variable, forcing nginx to refresh name lookups in the swarm.