Nginx doesn't get host from embedded DNS in docker 1.10.1

Question

I have several services running in docker containers, and one nginx server in front of them, all containers within the same docker network. Before docker 1.10 I created a dnsmasq container and within nginx set resolver to it so my services can be discovered from nginx.

But after upgrading to docker 1.10.1, the dnsmasq image doesn't work. After some research, I tried to use the new "embedded dns" from address 127.0.0.11 instead (which I think should be the correct way to use within docker network). Though I can ping to the service container from nginx container, but nginx keep sending back could not be resolved (3: Host not found).

And then I tried to use host command, I got very confusing result: (first ip address, and then host not found - same code as I got from nginx)

# host service-ui
service-ui has address 172.18.0.7
Host service-ui not found: 3(NXDOMAIN)
Host service-ui not found: 3(NXDOMAIN)

Ping works

/# ping service-ui
PING service-ui (172.18.0.7): 56 data bytes
64 bytes from 172.18.0.7: icmp_seq=0 ttl=64 time=1.123 ms
64 bytes from 172.18.0.7: icmp_seq=1 ttl=64 time=0.131 ms
64 bytes from 172.18.0.7: icmp_seq=2 ttl=64 time=0.178 ms

Following is a simplified version of my configurations:

docker-compose.yml

version: '2'

services:
  nginx:
    build: .
    ports:
      - 80:80   
  service-ui:
    image: service-ui
networks:
  default:
    external:
      name: ssx

nginx.conf within container

 location ~ /([a-z]*)/ {
  resolver 127.0.0.11;
  proxy_pass http://$1-ui:9000$uri$is_args$args;
}

Here's also the verbose result from host:

# host -v service-ui
Trying "service-ui"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55237
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;service-ui.            IN  A

;; ANSWER SECTION:
service-ui.     1800    IN  A   172.18.0.7

Received 52 bytes from 127.0.0.11#53 in 1 ms
Trying "service-ui"
Host service-ui not found: 3(NXDOMAIN)
Received 102 bytes from 127.0.0.11#53 in 1 ms
Received 102 bytes from 127.0.0.11#53 in 1 ms
Trying "service-ui"
Host service-ui not found: 3(NXDOMAIN)
Received 102 bytes from 127.0.0.11#53 in 1 ms
Received 102 bytes from 127.0.0.11#53 in 1 ms

My docker version

Client:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

Can anyone help me figure out what goes wrong? Or how to dig into this issue?

Answer 1

This is because the docker dns server does not do ipv6, but the nginx resolver queries for both ipv6 and ipv4 as of 1.5.8: http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver

You can disable ipv6 easily enough on the resolver line in nginx (see the link above, it's just ipv6=off) and then it should start working for you.