DNS not working within docker containers when host uses dnsmasq and Google's DNS server are firewalled?

Question

The symptom is: the host machine has proper network access, but programs running within containers can't resolve DNS names (which may appear to be "can't access the network" before investigating more).

$ sudo docker run -ti mmoy/ubuntu-netutils /bin/bash
root@082bd4ead733:/# ping www.example.com
... nothing happens (timeout) ... ^C
root@082bd4ead733:/# host www.example.com
... nothing happens (timeout) ... ^C

(The docker image mmoy/ubuntu-netutils is a simple image based on Ubuntu with ping and host included, convenient here since the network is broken and we can't apt install these tools)

The issue comes from the fact that docker automatically configured Google's public DNS as DNS server within the container:

root@082bd4ead733:/# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

nameserver 8.8.8.8
nameserver 8.8.4.4

This just works in many configurations, but obviously doesn't when the host runs on a network where Google's public DNS are filtered by some firewall rules.

The reason this happened is:

• Docker first tries configuring the same DNS server(s) on the host and within the container.
• The host runs dnsmasq, a DNS caching service. dnsmasq acts as a proxy for DNS requests, hence the apparent DNS server in the host's /etc/resolve.conf is nameserver 127.0.1.1, i.e. localhost.
• The host's dnsmasq listens only for requests comming from localhost and blocks requests coming from the docker container.
• Since using 127.0.1.1 within docker doesn't work, docker falls back to Google's public DNS, which do not work either.

There may be several reasons why DNS is broken within docker containers. This question (and answers) covers the case where:

• dnsmasq is used. To check whether this is the case:
  • Run ps -e | grep dnsmasq on the host. If the output is empty, you're not running dnsmasq.
  • Check the host's resolv.conf, it probably contains an entry like nameserver 127.0.1.1. If it contains nameserver 127.0.0.53, you're probably running systemd-resolved instead of dnsmasq. If so, you won't be able to use the solution forwading DNS requests to dnsmasq (the one using listen-address=172.17.0.1). systemd-resolved versions earlier than 247 hardcoded the fact that it listens only on the 'lo' interface hence there's no easy way to adapt this solution with these versions. Other answers below will work with systemd-resolved.
• Google's public DNS is filtered. Run host www.example.com 8.8.8.8. If it fails or times out, then you are in this situation.

What are the solutions to get a proper DNS configuration in this configuration?

Answer 1

Since the automatic DNS discovery is guilty here, you may override the default setting in docker's configuration.

First, get the IP of the DNS server dnsmasq is using with e.g.:

$ sudo kill -USR1 `pidof dnsmasq`
$ sudo tail /var/log/syslog 
[...]
Apr 24 13:20:19 host dnsmasq[2537]: server xx.yy.zz.tt1#53: queries sent 0, retried or failed 0
Apr 24 13:20:19 host dnsmasq[2537]: server xx.yy.zz.tt2#53: queries sent 0, retried or failed 0

The IP addresses correspond to the xx.yy.zz.tt placeholders above.

Alternatively, if your system is using systemd-resolve instead of dnsmasq, run:

$ resolvectl status | grep 'Current DNS'
Current DNS Server: xx.yy.zz.tt

You can set the DNS at docker run time with the --dns option:

$ sudo docker run --dns xx.yy.zz.tt1 --dns xx.yy.zz.tt2 -ti mmoy/ubuntu-netutils bash
root@6c5d08df5dfd:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=54 time=86.6 ms
64 bytes from 93.184.216.34: icmp_seq=2 ttl=54 time=86.6 ms

One advantage of this solution is that there is no configuration file involved, hence no risk of forgetting about the configuration and running into troubles later because of a specific config: you're getting this DNS configuration if and only if you type the --dns option.

A drawback is that you won't get any DNS caching in the containers, hence DNS resolution will be slower.

Alternatively you may set it permanently in Docker's configuration file, /etc/docker/daemon.json (create it, on the host, if it doesn't exist):

$ cat /etc/docker/daemon.json
{
    "dns": ["xx.yy.zz.tt1", "xx.yy.zz.tt2"]
}

You need to restart the docker daemon to take the daemon.json file into account:

sudo service docker restart

Then you can check the configuration:

$ sudo docker run -ti mmoy/ubuntu-netutils bash
root@56c74d3bd94b:/# cat /etc/resolv.conf 
nameserver xx.yy.zz.tt1
nameserver xx.yy.zz.tt2
root@56c74d3bd94b:/# ping www.example.com
PING www.example.com (93.184.216.34) 56(84) bytes of data.
64 bytes from 93.184.216.34: icmp_seq=1 ttl=54 time=86.5 ms

Note that this hardcodes the DNS IP in your configuration files. This is strongly discouraged if your machine is a laptop that connects to different networks, and may be problematic if your internet service provider changes the IP of the DNS servers.

Answer 2

One way is to use a user defined network for your container. In that case the container's /etc/resolv.conf will have the nameserver 127.0.0.11 (a.k.a. the Docker's embedded DNS server), which can forward DNS requests to the host's loopback address properly.

$ cat /etc/resolv.conf
nameserver 127.0.0.1
$ docker run --rm alpine cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
$ docker network create demo
557079c79ddf6be7d6def935fa0c1c3c8290a0db4649c4679b84f6363e3dd9a0
$ docker run --rm --net demo alpine cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0    

If you use docker-compose, it will set up a custom network for your services automatically (with a file format v2+). Note, however, that while docker-compose runs containers in a user-defined network, it still builds them in the default network. To use a custom network for builds you can specify the network parameter in the build configuration (requires file format v3.4+).

Answer 3

I just had to deal with this last night and eventually remembered that docker run has a set of options for handling it. I used --dns to specify the DNS server I want the container to use. Works like a champ and no need to hack my docker host. There are other options for the domain name and search suffixes.