Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Docker's `docker0` device dies repeatedly (`inet addr` disappears)

I'm running Docker version 1.4.1, build 5bc2ff8 on Ubuntu 14.04. When I docker run any container, after a few minutes my docker0 bridge "dies", and the container stops being able to reach the network. Before the connection dies, running ifconfig reports a docker0 device with an inet addr like:

docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99  
          inet addr:172.17.42.1  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: xxxx::xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          [... etc.]

But after the connection dies, ifconfig shows that the ipv4 address has gone away:

docker0   Link encap:Ethernet  HWaddr 56:84:7a:fe:97:99 
          inet6 addr: xxxx::xxxx:xxxx:xxxx:xxxx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8116 errors:0 dropped:0 overruns:0 frame:0
          TX packets:15995 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:2444859 (2.4 MB)  TX bytes:17440729 (17.4 MB)

Restarting docker, e.g. with sudo service docker restart, brings the device back up -- but all my containers die and the problem starts over again. I can't reliably get anything to run for more than a few minutes at a time. Not long enough to even complete a docker build for most projects.

  1. What could be causing this?
  2. How can I diagnose it?
  3. What are some possible solutions?

Thanks!


Update: I can reliably trigger this docker0-dropping behavior simply by starting a container with docker run -t -i ubuntu /bin/bash, and then exiting with ctrl-d. When I do so, here's what I see in /var/log/syslog

myhost kernel:  docker0: port 1(veth80ddeaf) entered disabled state
myhost kernel:  device veth80ddeaf left promiscuous mode
myhost kernel:  docker0: port 1(veth80ddeaf) entered disabled state
'

myhost dhclient: Internet Systems Consortium DHCP Client 4.2.4
myhost dhclient: Copyright 2004-2012 Internet Systems Consortium.
myhost dhclient: All rights reserved.
myhost dhclient: For info, please visit https://www.isc.org/software/dhcp/
myhost dhclient: 
myhost dhclient: Listening on LPF/docker0/56:84:7a:fe:97:99
myhost dhclient: Sending on   LPF/docker0/56:84:7a:fe:97:99
myhost dhclient: Sending on   Socket/fallback
myhost kernel:  IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready

Update #2: The frequency of failure seems to depend on how long the container runs. For example:

docker run -i -t  ubuntu   sleep 0
--> `docker0` "survives" ~100% of the time

docker run -i -t  ubuntu   sleep 1
--> `docker0` survives ~80% of the time

docker run -i -t  ubuntu   sleep 5
--> `docker0` survives ~0% of the time
like image 419
Bosh Avatar asked Mar 08 '15 06:03

Bosh


People also ask

What is docker0 in Ifconfig?

Whenever you run a docker container, a default bridge network call docker0 gets associated with the container unless any other network is specified. For example, when I run ifconfig command, you will get the details of docker0 network of bridge type along with other network details.

What is docker0 in Linux?

Docker is an open source project that automates the deployment of applications inside Linux Containers, and provides the capability to package an application with its runtime dependencies into a container.

What is docker0 IP?

Also, the “docker0” bridge has an IP address of 172.17. 0.1 and should be the default gateway for the host.

What is docker0 Ubuntu?

Docker is a container service which allows one to run applications or even operating systems on a host operating system as containers. Containers are a new and exciting technology that has evolved over the last couple of years and being adopted by a lot of key organizations.


1 Answers

How can I diagnose it?

When docker0 has an ip address, does it go away if you don't start any containers? If it persists indefinitely until you start a container, I would start by looking at the Docker logs as well as tailing the system logs when you start a container.

Does the ip address disappear at set intervals (e.g., every N minutes)? If so, I would look for logs from cron to see if some periodic task is responsible.

Are you running NetworkManager? Does disabling NetworkManager make the problem go away? I am running Docker on a system with NetworkManager without a problem, but I have no-auto-default=* set in my config, which may have an impact on this sort of thing.

Update

This is very suspicious:

myhost dhclient: Internet Systems Consortium DHCP Client 4.2.4
myhost dhclient: Copyright 2004-2012 Internet Systems Consortium.
myhost dhclient: All rights reserved.
myhost dhclient: For info, please visit https://www.isc.org/software/dhcp/
myhost dhclient: 
myhost dhclient: Listening on LPF/docker0/56:84:7a:fe:97:99
myhost dhclient: Sending on   LPF/docker0/56:84:7a:fe:97:99
myhost dhclient: Sending on   Socket/fallback

There should not be any dhclient process listening on docker0, and this is absolutely what is causing your ip address to disappear. If you are not explicitly running a dhcp client on this interface, this really suggests that NetworkManager is in fact trying to manage this interface. You said you disabled NetworkManager, but did you confirm that the process was stopped? What is the parent process of the dhclient that is listening on docker0? If you stop the dhclient process, does it get restarted? Does the problem go away?

like image 77
larsks Avatar answered Oct 22 '22 12:10

larsks