docker compose: rebuild of one linked container breaks nginx's upstream

Question

I'm using docker-compose with "Docker for Mac" and I have two containers: one NGINX, one container serving a node-app on port 3000.

docker-compose.yml looks like this:

version: "2"

services:
  nginx:
    build: ./nginx
    ports:
      - "80:80"
    links:
      - api
  api:
    build: ./api
    volumes:
      - "./api:/opt/app"

In the NGINX's config I say:

upstream api {
  server api:3000;
}

server {
  # ....
  location ~ ^/api/?(.*) {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;

    proxy_pass http://api;
    proxy_redirect off;
  }
}

Now, when I change something in the node code and rebuild the container

$ docker-compose stop api && docker-compose up -d --build --no-deps api

the container is getting rebuilt and started. The problem is, that sometimes the internal IP of the container changes and NGINX won't know about that. Funny enough, when I go into the NGINX container and ping api I get the new IP address

$ ping api
PING api (172.19.0.3): 56 data bytes
64 bytes from 172.19.0.3: icmp_seq=0 ttl=64 time=0.236 ms

but NGINX logs still say

2016/10/20 14:20:53 [error] 9#9: *9 connect() failed (113: No route to host) while connecting to upstream, client: 172.19.0.1, server: localhost, request: "GET /api/test HTTP/1.1", upstream: "http://172.19.0.7:3000/api/test", host: "localhost"

where the upstream's 172.19.0.7 is still the old IP address.

PS: this doesn't happen every time I rebuild the container.

Answer 1

This is because Nginx caches the DNS response for upstream servers - in your workflow you're only restarting the app container, so Nginx doesn't reload and always uses its cached IP address for the api container.

When you run a new api container, as you've seen, it can have a different IP address so the cache in Nginx is not valid. The ping works because it doesn't cache Docker's DNS response.

Assuming this is just for dev and downtime isn't an issue, docker-compose restart nginx after you rebuild the app container will restart Nginx and clear the DNS cache.