`vagrant up` times out at "Waiting for domain to get an IP address..."

Question

This appears to only happen when I'm using the generic/arch box. I've tried several ubuntu boxes and everything works fine.

Host OS is Manjaro.

It's freezing with output:

 INFO interface: info: ==> default: Waiting for domain to get an IP address...                                                                   
==> default: Waiting for domain to get an IP address...
 INFO retryable: Retryable exception raised: #<Fog::Errors::TimeoutError: The specified wait_for timeout (2 seconds) was exceeded>

libvirt/virsh version 3.10.0 vagrant version 2.0.1

OS 4.9.76-1-MANJARO

Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "generic/arch"
end

Below is what I think the relevant output is from VAGRANT_LOG=debug vagrant up

==> default: Creating shared folders metadata...                                                                                                               
 INFO warden: Calling IN action: #<VagrantPlugins::ProviderLibvirt::Action::CreateNetworks:0x0000560dfaea3450>                                                 
 INFO create_networks: Using vagrant-libvirt at 192.168.121.0/24 as the management network nat is the mode                                                     
DEBUG create_networks: In config found network type forwarded_port options {:guest=>22, :host=>2222, :host_ip=>"127.0.0.1", :id=>"ssh", :auto_correct=>true, :p
rotocol=>"tcp"}                                                                                                                                                
DEBUG create_networks: Searching for network with options {:iface_type=>:private_network, :network_name=>"vagrant-libvirt", :ip=>"192.168.121.0", :netmask=>"25
5.255.255.0", :dhcp_enabled=>true, :forward_mode=>"nat", :guest_ipv6=>"yes", :autostart=>false}                                                                
/home/eihli/.vagrant.d/gems/2.4.3/gems/nokogiri-1.6.8.1/lib/nokogiri/xml/document.rb:44: warning: constant ::Fixnum is deprecated                              
DEBUG create_networks: looking up network with ip == 192.168.121.0                                                                                             
DEBUG create_networks: Checking that network name does not clash with ip                                                                                       
DEBUG create_networks: looking up network named vagrant-libvirt                                                                                                
DEBUG create_networks: generating name for bridge                                                                                                              
DEBUG create_networks: looking up bridge named virbr0                                                                                                          
DEBUG create_networks: looking up bridge named virbr1                                                                                                          
DEBUG create_networks: found available bridge name virbr1                                                          
DEBUG create_networks: created network                                                        
 INFO create_networks: Saving information about created network vagrant-libvirt, UUID=71c9f6dd-4b2d-48f2-a787-5a0439d2e587 to file /home/eihli/code/vagrant_te$
t/.vagrant/machines/default/libvirt/created_networks.                                                             
 INFO warden: Calling IN action: #<VagrantPlugins::ProviderLibvirt::Action::CreateNetworkInterfaces:0x0000560dfb42cae8>
 INFO create_network_interfaces: Using vagrant-libvirt at 192.168.121.0/24 as the management network nat is the mode                                           
DEBUG create_network_interfaces: In config found network type forwarded_port options {:guest=>22, :host=>2222, :host_ip=>"127.0.0.1", :id=>"ssh", :auto_correc$
=>true, :protocol=>"tcp"}                                                                                                                                      
DEBUG create_network_interfaces: Adapter not specified so found slot 0                                                                                         
DEBUG create_network_interfaces: Found network by name                                                                                                         
 INFO create_network_interfaces: Creating network interface eth0 connected to network vagrant-libvirt.                                                         
 INFO warden: Calling IN action: #<VagrantPlugins::ProviderLibvirt::Action::SetBootOrder:0x0000560dfb30af70>                                                  
 INFO warden: Calling IN action: #<VagrantPlugins::ProviderLibvirt::Action::StartDomain:0x0000560dfafc46e0>                                                    
 INFO interface: info: Starting domain.                                                                                                                        
 INFO interface: info: ==> default: Starting domain.                                                                                                           
==> default: Starting domain.                                                                                                                                  
 INFO warden: Calling IN action: #<VagrantPlugins::ProviderLibvirt::Action::WaitTillUp:0x0000560dfab82de8>
DEBUG wait_till_up: Searching for IP for MAC address: 52:54:00:62:61:1d                                     
 INFO interface: info: Waiting for domain to get an IP address...                                                        
 INFO interface: info: ==> default: Waiting for domain to get an IP address...                                                                   
==> default: Waiting for domain to get an IP address...  

I checked wireshark using tshark -i virbr1 and I don't get any output. I wasn't sure which device I should be listening to though, so I also checked on tshark -i vnet0 and I did get some output but nothing that looked DHCP related.

Below is the output of tshark -i vnet0

Capturing on 'vnet0'
1 0.000000000 fe80::fc54:ff:fe45:368d → ff02::fb     MDNS 201 Standard query response 0x0000 PTR, cache flush Ophelia.local AAAA, cache flush fe80::fc54:ff:fe45:368d
2 0.745989853 fe:54:00:45:36:8d → Spanning-tree-(for-bridges)_00 STP 52 Conf. TC + Root = 32768/0/52:54:00:78:b4:d1  Cost = 0  Port = 0x8002
3 1.599338528 fe80::fc54:ff:fe45:368d → ff02::2      ICMPv6 70 Router Solicitation from fe:54:00:45:36:8d
4 2.190388300 fe80::fc54:ff:fe45:368d → ff02::fb     MDNS 201 Standard query response 0x0000 PTR, cache flush Ophelia.local AAAA, cache flush fe80::fc54:ff:fe45:368d
5 2.666024617 fe:54:00:45:36:8d → Spanning-tree-(for-bridges)_00 STP 52 Conf. TC + Root = 32768/0/52:54:00:78:b4:d1  Cost = 0  Port = 0x8002
6 4.799357852 fe:54:00:45:36:8d → Spanning-tree-(for-bridges)_00 STP 52 Conf. TC + Root = 32768/0/52:54:00:78:b4:d1  Cost = 0  Port = 0x8002

Below is the result of dmesg -w from the time when I vagrant up to the time I ctrl-c

[ 2048.355424] virbr1: port 2(vnet0) entered disabled state
[ 2048.358280] device vnet0 left promiscuous mode
[ 2048.358287] virbr1: port 2(vnet0) entered disabled state
[ 2048.578278] device virbr1-nic left promiscuous mode
[ 2048.578287] virbr1: port 1(virbr1-nic) entered disabled state
[ 2059.394512] virbr1: port 1(virbr1-nic) entered blocking state
[ 2059.394516] virbr1: port 1(virbr1-nic) entered disabled state
[ 2059.394598] device virbr1-nic entered promiscuous mode
[ 2059.532014] virbr1: port 1(virbr1-nic) entered blocking state
[ 2059.532017] virbr1: port 1(virbr1-nic) entered listening state
[ 2059.567010] virbr1: port 1(virbr1-nic) entered disabled state
[ 2059.694140] virbr1: port 2(vnet0) entered blocking state
[ 2059.694156] virbr1: port 2(vnet0) entered disabled state
[ 2059.694377] device vnet0 entered promiscuous mode
[ 2059.695080] virbr1: port 2(vnet0) entered blocking state
[ 2059.695083] virbr1: port 2(vnet0) entered listening state
[ 2061.871278] virbr1: port 2(vnet0) entered learning state
[ 2064.004553] virbr1: port 2(vnet0) entered forwarding state
[ 2064.004555] virbr1: topology change detected, propagating
[ 2065.750798] kvm [5143]: vcpu0, guest rIP: 0xffffffff86061ce6 unhandled rdmsr: 0x140
[ 2065.821598] kvm [5143]: vcpu1, guest rIP: 0xffffffff86061ce6 unhandled rdmsr: 0x140
[ 2065.861278] kvm: zapping shadow pages for mmio generation wraparound
[ 2065.863039] kvm: zapping shadow pages for mmio generation wraparound
[ 2066.017675] kvm [5143]: vcpu0, guest rIP: 0xffffffff86061ce6 unhandled rdmsr: 0x34
[ 2068.941350] virbr1: port 2(vnet0) entered disabled state
[ 2068.943507] device vnet0 left promiscuous mode
[ 2068.943511] virbr1: port 2(vnet0) entered disabled state
[ 2069.163524] device virbr1-nic left promiscuous mode
[ 2069.163532] virbr1: port 1(virbr1-nic) entered disabled state

Below is the output of ip addr show (with the exception that this is the umpteenth time I've run vagrant up and each time the number of the virtual network interface goes up.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host 
   valid_lft forever preferred_lft forever
2: wlp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 3c:a9:f4:4c:7b:3c brd ff:ff:ff:ff:ff:ff
inet 10.7.74.120/18 brd 10.7.127.255 scope global dynamic noprefixroute wlp3s0
   valid_lft 599674sec preferred_lft 599674sec
inet6 fe80::3d00:e921:d2e6:d9d5/64 scope link noprefixroute 
   valid_lft forever preferred_lft forever
3: enp0s25: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
link/ether 28:d2:44:17:62:46 brd ff:ff:ff:ff:ff:ff
34: virbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:ff:e1:a6 brd ff:ff:ff:ff:ff:ff
inet 192.168.121.1/24 brd 192.168.121.255 scope global virbr1
   valid_lft forever preferred_lft forever
35: virbr1-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr1 state DOWN group default qlen 1000
link/ether 52:54:00:ff:e1:a6 brd ff:ff:ff:ff:ff:ff
36: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr1 state UNKNOWN group default qlen 1000
link/ether fe:54:00:81:4a:63 brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe81:4a63/64 scope link 
   valid_lft forever preferred_lft forever

Answer 1

In my case, I had overly restrictive iptables rules preventing the NAT which is set up for the virtual network bridge(s) virbr* by Vagrant via libvirt.

As an experiment, I disabled all firewall rules before starting the Vagrant machine and it was able to get an IP address.

Answer 2

Since you are using the libvirt provider, you should be able to check out the state of the virtual machine with virt-manager.

The rest of the answer is written under the two assumptions that my system is not so different from yours and that I have run into the same problem using vagrant-libvirt and generic/arch (v1.3.40)

Opening the VM in virt-manager, I've found out that the system cannot detect the root filesystem and also boots into an emergency shell. That's because the initramfs-linux.img provided by the generic/arch box does not include the necessary kernel modules for the default settings that the libvirt-provider with QEMU/kvm uses.

If it's somehow important to boot this box, you should be able to start the box by using the fallback image because it ships more modules. To make it rebootable without intervention, you can run sudo mkinitcpio -p linux to add the missing modules in the guest. Alternatively, you could also change the SCSI controller from Hypervisor Defaults to VirtIO SCSI (eg. with virt-manager) on the host.

For a more permanent solution (nothing of the aforementioned would survive a vagrant destroy), there ist not much to be done - besides maybe waiting for a "fixed" release of the box or patching the plugin.

ps. The easiest solution would probably be to pick another box (e.g. archlinux/archlinux).