Can't create docker network: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Question

Commmand docker create network mynetwork gave me following error response:

Error response from daemon: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

I've googled this error but none of these solutions work:

1. I have only 3 pre-defined docker network, so docker network prune makes no difference.
2. I have never installed OpenVPN

Here is some info about my server's environment. OS:

LSB Version:    :core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description:    CentOS release 6.3 (Final)
Release:    6.3
Codename:   Final

kernel:

Linux xxx.com 3.10.0_3-0-0-17 #181 SMP Thu Feb 8 16:34:08 CST 2018 x86_64 x86_64 x86_64 GNU/Linux

docker:

Client:
 Version:   18.03.0-ce
 API version:   1.37
 Go version:    go1.9.2
 Git commit:    0520e24
 Built: Wed Mar 21 23:05:52 2018
 OS/Arch:   linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:  18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:    Wed Mar 21 23:14:54 2018
  OS/Arch:  linux/amd64
  Experimental: false

The log output from Docker daemon when executing docker network create:

time="2018-04-02T14:46:15.280752033+08:00" level=debug msg="Calling POST /v1.37/networks/create"
time="2018-04-02T14:46:15.280955481+08:00" level=debug msg="form data: {\"Attachable\":false,\"CheckDuplicate\":true,\"ConfigFrom\":null,\"ConfigOnly\":false,\"Driver\":\"bridge\",\"EnableIPv6\":false,\"IPAM\":{\"Config\":[],\"Driver\":\"default\",\"Options\":{}},\"Ingress\":false,\"Internal\":false,\"Labels\":{},\"Name\":\"abcd\",\"Options\":{},\"Scope\":\"\"}"
time="2018-04-02T14:46:15.281417190+08:00" level=debug msg="Allocating IPv4 pools for network abcd (0b9717a7272a4c8782c49fa566d7e6a06704c0d085e6389e942998fbe97462a6)"
time="2018-04-02T14:46:15.281444055+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.281485165+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.281508839+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.281710387+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.281736496+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.281750737+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.281845693+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.281870666+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.281884536+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.281978263+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.282003622+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282017850+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282100720+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.282126269+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282139742+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282231487+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.282257877+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282271266+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282400372+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.282428023+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282442208+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282535390+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.282567708+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282581458+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282665177+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.282693288+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282706401+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282801403+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.282830506+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282843584+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282930538+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.282960034+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.282972907+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283059526+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.283088441+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283101529+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283189181+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.283224740+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283238227+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283332740+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.283365026+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283379846+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283460062+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.283491194+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283504728+08:00" level=debug msg="Received set for ordinal 65535, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283588953+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.283622415+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283643410+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283729518+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.283765598+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283780273+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283859445+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.283893519+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283909058+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.283993771+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.284028573+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284041138+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284128998+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.284170756+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284185810+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284263887+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.284303373+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284316600+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284397202+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.284436789+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284449960+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284526967+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.284567439+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284580543+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284662281+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.284703475+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284716169+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284794971+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.284835314+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284848272+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284926857+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.284976293+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.284996202+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.285094175+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.285155415+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.285171491+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.285267913+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.285314698+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.285327856+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.285419681+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.285463945+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.285476909+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.285565520+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.285614270+08:00" level=debug msg="Received set for ordinal 0, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.285631714+08:00" level=debug msg="Received set for ordinal 4095, start 0, end 0, any false, release false, serial:false curr:0 \n"
time="2018-04-02T14:46:15.285716032+08:00" level=debug msg="RequestPool(LocalDefault, , , map[], false)"
time="2018-04-02T14:46:15.285760262+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.240.0/20)"
time="2018-04-02T14:46:15.285778212+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.224.0/20)"
time="2018-04-02T14:46:15.285790635+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.208.0/20)"
time="2018-04-02T14:46:15.285802203+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.192.0/20)"
time="2018-04-02T14:46:15.285813998+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.176.0/20)"
time="2018-04-02T14:46:15.285825543+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.160.0/20)"
time="2018-04-02T14:46:15.285836927+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.144.0/20)"
time="2018-04-02T14:46:15.285848868+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.128.0/20)"
time="2018-04-02T14:46:15.285860012+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.112.0/20)"
time="2018-04-02T14:46:15.285871053+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.80.0/20)"
time="2018-04-02T14:46:15.285882613+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.64.0/20)"
time="2018-04-02T14:46:15.285894331+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.48.0/20)"
time="2018-04-02T14:46:15.285906570+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.32.0/20)"
time="2018-04-02T14:46:15.285917790+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.16.0/20)"
time="2018-04-02T14:46:15.285930785+08:00" level=debug msg="ReleasePool(LocalDefault/192.168.0.0/20)"
time="2018-04-02T14:46:15.285942133+08:00" level=debug msg="ReleasePool(LocalDefault/172.31.0.0/16)"
time="2018-04-02T14:46:15.285953628+08:00" level=debug msg="ReleasePool(LocalDefault/172.30.0.0/16)"
time="2018-04-02T14:46:15.285965120+08:00" level=debug msg="ReleasePool(LocalDefault/172.29.0.0/16)"
time="2018-04-02T14:46:15.285975972+08:00" level=debug msg="ReleasePool(LocalDefault/172.28.0.0/16)"
time="2018-04-02T14:46:15.285986956+08:00" level=debug msg="ReleasePool(LocalDefault/172.27.0.0/16)"
time="2018-04-02T14:46:15.285997910+08:00" level=debug msg="ReleasePool(LocalDefault/172.26.0.0/16)"
time="2018-04-02T14:46:15.286015087+08:00" level=debug msg="ReleasePool(LocalDefault/172.25.0.0/16)"
time="2018-04-02T14:46:15.286026575+08:00" level=debug msg="ReleasePool(LocalDefault/172.24.0.0/16)"
time="2018-04-02T14:46:15.286037792+08:00" level=debug msg="ReleasePool(LocalDefault/172.23.0.0/16)"
time="2018-04-02T14:46:15.286049714+08:00" level=debug msg="ReleasePool(LocalDefault/172.22.0.0/16)"
time="2018-04-02T14:46:15.286063811+08:00" level=debug msg="ReleasePool(LocalDefault/172.21.0.0/16)"
time="2018-04-02T14:46:15.286076360+08:00" level=debug msg="ReleasePool(LocalDefault/172.20.0.0/16)"
time="2018-04-02T14:46:15.286087965+08:00" level=debug msg="ReleasePool(LocalDefault/172.19.0.0/16)"
time="2018-04-02T14:46:15.286099514+08:00" level=debug msg="ReleasePool(LocalDefault/172.18.0.0/16)"
time="2018-04-02T14:46:15.286110733+08:00" level=debug msg="ReleasePool(LocalDefault/172.17.0.0/16)"

Appreciate your help!

Answer 1

the defaults part of the error is referring to the 31 predetermined address ranges that docker is using unless you specify --subnet option when running the command (in which case the error would go away):

$ docker network create abcd --subnet 172.17.0.0/16

The log output from your Docker deamon indicates that it's actually trying to allocate almost each of those 31 default subnets. There's only two possible reasons why that allocation may fail.

Name server overlap

Docker won't use a default subnet if your DNS server belongs to that subnet. Obviously, it's unlikely to be the issue, but we can still reproduce your error by configuring some (dummy) name servers, one per each of the 31 default subnet:

$ cat /etc/resolv.conf
nameserver 172.17.0.1
nameserver 172.18.0.1
nameserver 172.19.0.1
nameserver 172.20.0.1
nameserver 172.21.0.1
nameserver 172.22.0.1
nameserver 172.23.0.1
nameserver 172.24.0.1
nameserver 172.25.0.1
nameserver 172.26.0.1
nameserver 172.27.0.1
nameserver 172.28.0.1
nameserver 172.29.0.1
nameserver 172.30.0.1
nameserver 172.31.0.1
nameserver 192.168.0.1
nameserver 192.168.16.1
nameserver 192.168.32.1
nameserver 192.168.48.1
nameserver 192.168.64.1
nameserver 192.168.80.1
nameserver 192.168.96.1
nameserver 192.168.112.1
nameserver 192.168.128.1
nameserver 192.168.144.1
nameserver 192.168.160.1
nameserver 192.168.176.1
nameserver 192.168.192.1
nameserver 192.168.208.1
nameserver 192.168.224.1
nameserver 192.168.240.1

$ docker network create abcd
Error response from daemon: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Ubuntu users reading this can do the same with /run/systemd/resolve/resolv.conf.

IP route overlap

Docker official docs point to an article (UPD: cached page) which says

By default bridge is assigned one subnet from the ranges 172.[17-31].0.0/16 or 192.168.[0-256].0/20 which does not overlap with any existing host interface.

Which means, just like with name servers, we could have your error reproduced by adding some (dummy) routes matching all the 31 default subnets to our routing table:

$ ip route add 172.16.0.0/12  dev lo
$ ip route add 192.168.0.0/16 dev lo

$ docker network create abcd
Error response from daemon: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

Solving IP route overlap

In your case, most likely the output of ip route needs to be analyzed with two purposes in mind:

1. Find and delete unnecessary routes conflicting with 172.[17-31].0.0/16 or 192.168.[0-256].0/20
2. Or find unused IP range to use with --subnet option. That range should lie within private networks 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16 as per RFC-1918, if you host is connected to the Internet

---

The rest is irrelevant to OP but may be helpful for users coming from search

OpenVPN

Having option --redirect-gateway def1 set (or pushed from the VPN server), OpenVPN will add IP routes 0.0.0.0/1 and 128.0.0.0/1 to the routing table. And because these two subnets overlap with any possible subnet, that will cause the error in question.

It can be avoided by using --redirect-gateway (without a flag) option instead, (combined with --pull-filter ignore "redirect-gateway def1" if the original option gets pushed from the VPN server).

Docker Compose

Here's (roughly speaking) the equivalent of --subnet 172.17.0.0/16 option in docker-compose.yml:

networks:
  default:
    ipam:
      config:
      - subnet: 172.17.0.0/16