How can I give Docker containers access to a dnsmasq local DNS resolver on the host?

Question

There are lots of ways in which Docker containers can get confused about DNS settings (just search SO or the wider internet for "Docker DNS" to see what I mean), and one of the common workarounds suggested is to:

1. Set up dnsmasq as a local DNS resolver on the host system
2. Bind it to the docker0 network interface
3. Configure Docker to use the docker0 IP address for DNS resolution

However, attempting to apply this workaround naively on many modern Linux systems will send you down a rabbithole of Linux networking and process management complexity, as systemd assures you that dnsmasq isn't running, but netstat tells you that it is, and actually attempting to start dnsmasq fails with the complaint that port 53 is already in use.

So, how do you reliably give your containers access to a local resolver running on the host, even if the system already has one running by default?

Answer 1

The problem here is that many modern Linux systems run dnsmasq implicitly, so what you're now aiming to do is to set up a second instance specifically for Docker to use. There are actually 3 settings needed to do that correctly:

• --interface=docker0 to listen on the default Docker network interface
• --except-interface=lo to skip the implicit addition of the loopback interface
• --bind-interfaces to turn off a dnsmasq feature where it still listens on all interfaces by default, even when its only processing traffic for one of them

Setting up a dedicated dnsmasq instance

Rather than changing the settings of the default system wide dnsmasq instance, these instructions show setting up a dedicated dnsmasq instance with systemd, on a system which already defines a default dnsmasq service:

$ sudo cp /usr/lib/systemd/system/dnsmasq.service /etc/systemd/system/dnsmasq-docker.service $ sudoedit /etc/systemd/system/dnsmasq-docker.service 

First, we copy the default service settings to a dedicated service file. We then edit that service file, and look for the service definition section, which should be something like this:

[Service] ExecStart=/usr/sbin/dnsmasq -k 

We edit that section to define our additional options:

[Service] ExecStart=/usr/sbin/dnsmasq -k --interface=docker0 --except-interface=lo --bind-interfaces 

The entire file is actually pretty short:

[Unit] Description=DNS caching server. After=network.target After=docker.service Wants=docker.service  [Service] ExecStart=/usr/sbin/dnsmasq -k --interface=docker0 --except-interface=lo --bind-interfaces  [Install] WantedBy=multi-user.target 

The [Unit] section tells systemd to wait until after both the network stack and the main docker daemon are available to start this service, while [Install] indicates which system state target to add the service to when enabling it.

We then configure our new service to start on system boot, and also start it explicitly for immediate use:

$ sudo systemctl enable dnsmasq-docker $ sudo systemctl start dnsmasq-docker 

As the final step in getting the service running, we check it has actually started as expected:

$ sudo systemctl status dnsmasq-docker 

The two key lines we're looking for in that output are:

Loaded: loaded (/etc/systemd/system/dnsmasq-docker.service; enabled; vendor preset: disabled) Active: active (running) since <date & time> 

On the first line, note the "enabled" status, while on the second, the "active (running)" status. If the service hasn't started correctly, then the additional diagnostic information will hopefully explain why (although it can be unfortunately cryptic at times, hence this post).

Note: This configuration may fail to start dnsmasq-docker on system restart with an error about the docker0 interface not being defined. While waiting for docker.service seems to be pretty reliable in avoiding that problem, if name resolution from docker containers isn't working after a system restart, then try running:

$ sudo systemctl start dnsmasq-docker 

Configuring the host firewall

To be able to use the resolver from local Docker containers, we also need to drop the network firewall between the host and systems running in containers:

sudo firewall-cmd --permanent --zone=trusted --change-interface=docker0 sudo firewall-cmd --reload 

(This would be an absolutely terrible idea on a production container host, but can be a helpful risk-vs-convenience trade-off on a developer workstation)

Configuring Docker using a systemd environment file

Now that we have our local resolver running, we need to configure Docker to use it by default. Docker needs the IP address of the docker0 interface rather than the interface name, so we use ifconfig to retrieve that:

$ ifconfig docker0 | grep inet         inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0 

So, for my system, the host's interface on the default docker0 bridge is accessible as 172.17.0.1 (Appending | cut -f 10 -d ' ' to that command should filter the output to just the IP address)

Since I'm assuming a systemd-based Linux with a system provided Docker package, we'll query the system package's service file to find out how the service is being started:

$ cat /usr/lib/systemd/system/docker.service 

The first thing we're looking for is the exact command used to start the daemon, which should look something like this:

ExecStart=/usr/bin/docker daemon \           $OPTIONS \           $DOCKER_STORAGE_OPTIONS \           $DOCKER_NETWORK_OPTIONS \           $INSECURE_REGISTRY 

The second part we're looking for is whether or not the service is configured to use an environment file, as indicated by one of more lines like this:

EnvironmentFile=-/etc/sysconfig/docker 

When an environment file is in use (as it is on Fedora 23), then the way to change the Docker daemon settings is to edit that file and update the relevant environment variable:

$ sudoedit /etc/sysconfig/docker 

The existing OPTIONS entry on Fedora 23 looks like this:

OPTIONS='--selinux-enabled --log-driver=journald' 

To change the default DNS resolution settings, we amend it to look like this:

OPTIONS='--selinux-enabled --log-driver=journald --dns=172.17.0.1' 

And then restart the Docker daemon:

$ sudo systemctl restart docker 

With this change implemented, Docker containers should now be reliably able to access any systems your host system can access (including via VPN tunnels, which was my own reason for needing to figure this out)

You can run curl inside a container to check name resolution is working correctly:

docker run -it centos curl google.com 

Replace google.com with whichever hostname was giving you problems (as you should have only ended up finding this answer if you had a name resolution problem when running a process inside a Docker container)

Configuring Docker using a systemd drop-in file

(Caveat: since my system uses an environment file, I haven't been able to test the drop-in file based approach below, but it should work - I've included it since the Docker documentation seems to indicate they now prefer the use of systemd drop-in files to the use of environment files)

If the system service file doesn't use EnvironmentFile, then the entire ExecStart entry can be replaced by using a drop-in configuration file:

$ sudo mkdir -p /etc/systemd/system/docker.service.d $ sudoedit /etc/systemd/system/docker.service.d/daemon.conf 

We then tell Docker to clear the existing ExecStart entry and replace it with our new one with the additional settings:

[Service] ExecStart= ExecStart=/usr/bin/docker daemon \           $OPTIONS \           --dns 172.17.0.1 \           $DOCKER_STORAGE_OPTIONS \           $DOCKER_NETWORK_OPTIONS \           $INSECURE_REGISTRY 

We then tell systemd to load that configuration change and restart Docker:

$ sudo systemctl daemon-reload $ sudo systemctl restart docker 

References:

• Docker systemd config reference: https://docs.docker.com/engine/admin/systemd/
• systemd service file reference: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
• dnsmasq reference: http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html
• firewalld reference: https://fedoraproject.org/wiki/FirewallD
• Setting up dnsmasq without an existing local resolver on the host: http://docs.blowb.org/setup-host/dnsmasq.html

Answer 2

You can use the host's local DNS resolver (e.g. dnsmasq) from your Docker containers if they are on a custom network. In that case a container's /etc/resolv.conf will have the nameserver 127.0.0.11 (a.k.a. the Docker's embedded DNS server), which can forward DNS requests to the host's loopback address properly.

$ cat /etc/resolv.conf nameserver 127.0.0.1 $ docker run --rm alpine cat /etc/resolv.conf nameserver 8.8.8.8 nameserver 8.8.4.4 $ docker network create demo 557079c79ddf6be7d6def935fa0c1c3c8290a0db4649c4679b84f6363e3dd9a0 $ docker run --rm --net demo alpine cat /etc/resolv.conf nameserver 127.0.0.11 options ndots:0     

If you use docker-compose, it will set up a custom network for your services automatically (with a file format v2+). Note, however, that while docker-compose runs containers in a user-defined network, it still builds them in the default bridge network. To use a custom network for builds you can specify the network parameter in the build configuration (requires file format v3.4+).