cannot access Internet inside docker container when docker0 has non-default address

Question

Problem: the Internet isn't accessible within a docker container.

on my bare metal Ubuntu 17.10 box...

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=52 time=10.8 ms

but...

$ docker run --rm debian:latest ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
92 bytes from 7911d89db6a4 (192.168.220.2): Destination Host Unreachable

I think the root cause is that I had to set up a non-default network for docker0 because the default one 172.17.0.1 was already in use within my organization.

My /etc/docker/daemon.json file needs to look like this in order for docker to start successfully.

$ cat /etc/docker/daemon.json 
{
  "bip": "192.168.220.1/24",
  "fixed-cidr": "192.168.220.0/24",
  "fixed-cidr-v6": "0:0:0:0:0:ffff:c0a8:dc00/120",
  "mtu": 1500,
  "default-gateway": "192.168.220.10",
  "default-gateway-v6": "0:0:0:0:0:ffff:c0a8:dc0a",
  "dns": ["10.0.0.69","10.0.0.70","10.1.1.11"],
  "debug": true
}

Note that the default-gateway setting looks wrong. However, if I correct it to read 192.168.220.1 the docker service fails to start. Running dockerd at the command line directly produces the most helpful logging, thus:

With "default-gateway": 192.168.220.1 in daemon.json...

$ sudo dockerd
-----8<-----
many lines removed
----->8-----
Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to allocate secondary ip address (DefaultGatewayIPv4:192.168.220.1): Address already in use

Here's the info for docker0...

$ ip addr show docker0
10: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:10:bc:66:fd brd ff:ff:ff:ff:ff:ff
    inet 192.168.220.1/24 brd 192.168.220.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:10ff:febc:66fd/64 scope link 
       valid_lft forever preferred_lft forever

And routing table...

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.62.131.1     0.0.0.0         UG    100    0        0 enp14s0
10.62.131.0     0.0.0.0         255.255.255.0   U     100    0        0 enp14s0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 enp14s0
192.168.220.0   0.0.0.0         255.255.255.0   U     0      0        0 docker0

Is this the root cause? How do I achieve the, seemingly mutually exclusive states of:

• docker0 interface address is x.x.x.1
• gateway address is same, x.x.x.1
• dockerd runs ok

?

Thanks!

Longer answer to Wedge Martin's question. I made the changes to daemon.json as you suggested:

{
  "bip": "192.168.220.2/24",
  "fixed-cidr": "192.168.220.0/24",
  "fixed-cidr-v6": "0:0:0:0:0:ffff:c0a8:dc00/120",
  "mtu": 1500,
  "default-gateway": "192.168.220.1",
  "default-gateway-v6": "0:0:0:0:0:ffff:c0a8:dc0a",
  "dns": ["10.0.0.69","10.0.0.70","10.1.1.11"],
  "debug": true
}

so at least the daemon starts, but I still don't have internet access within a container...

$ docker run -it --rm debian:latest bash
root@bd9082bf70a0:/# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
15: eth0@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:c0:a8:dc:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.220.3/24 brd 192.168.220.255 scope global eth0
       valid_lft forever preferred_lft forever
root@bd9082bf70a0:/# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
92 bytes from bd9082bf70a0 (192.168.220.3): Destination Host Unreachable

Answer 1

It turned out that less is more. Simplifying daemon.json to the following resolved my issues.

{
  "bip": "192.168.220.2/24"
}