Vagrant multi-machine provisioning

Question

I'm trying to create multi-machine environment in Vagrant using Ansible as provisioner.

My Vagrantfile looks like next:

   Vagrant.configure("2") do |config|

    config.vm.provision "ansible" do |ansible|
       ansible.limit = "all"
       ansible.playbook = "main.yml"
       ansible.inventory_path = "staging"
       ansible.verbose = "-vvvv"
     end

    config.vm.define "machine1" do |machine1| 
        machine1.vm.box = "ubuntu/trusty64"
        machine1.vm.network "private_network", ip:"192.168.77.10"
        machine1.vm.hostname = "machine1"
        machine1.vm.provider :virtualbox do |vb|
           vb.name = "machine1"
        end
    end    

    config.vm.define "machine2" do |machine2| 
        machine2.vm.box = "ubuntu/trusty64"
        machine2.vm.network "private_network", ip:"192.168.77.11"
        machine2.vm.hostname = "machine2"
        machine2.vm.provider :virtualbox do |vb|
            vb.name = "machine2"
        end
    end    

    config.vm.define "machine3" do |machine3| 
        machine3.vm.box = "ubuntu/trusty64"
        machine3.vm.network "private_network", ip:"192.168.77.12"
        machine3.vm.hostname = "machine3"
        machine3.vm.provider :virtualbox do |vb|
           vb.name = "machine3"
        end
    end      
end

Inventory:

[AppServers]
192.168.77.10
192.168.77.11
192.168.77.12

[WebServers]
192.168.77.11
192.168.77.12

[WebLoadBalancers]
192.168.77.10

[SlaveDbServers]
192.168.77.10
192.168.77.12

[MasterDbServers]
192.168.77.11

[DbLoadBalancers]
192.168.77.11

main.yml:

- hosts: all
  roles:
  - Common
  - ConsulServer
  - ConsulAgent  

- hosts: WebServers
  roles:
  - WebServer

- hosts: WebLoadBalancers
  roles:
  - LoadBalancer

- hosts: MasterDbServers
  roles:
  - DbServer

I want to get 3 machines. All of them have to contain common soft(Consul servers and agents, vim etc). And some additional - own for each machine. But once i'm running "vagrant up" only first machine created, provisioner runs, fails because other 2 not created. Is it possible to run provisioner after all machines created? Or my approach is incorrect and i should perform this in other way? Thank you for your time.

Answer 1

The first problem I had was ERROR: cannot find role in.... I'm assuming you have these roles and excluded them for brevity. My advice here is to have a simple Ansible playbook when you are testing this:

---
- hosts: all
  gather_facts: false
  tasks:
  - command: hostname -f

Secondly, the problem at hand surround the use of static inventory file and caveats therein. You are seeing an error because the Ansible provisioner is failing to find all hosts when it runs after the first machine is up but the others are not.

Lastly, each machine will have a different key, which you must pass. So, following Vagrant's documented approach for multi-machine parallelism with Ansible and with help from this work-around, here is what I recommend your Vagrantfile look like:

Vagrant.configure("2") do |config|
  N = 3

  VAGRANT_VM_PROVIDER = "virtualbox"
  ANSIBLE_RAW_SSH_ARGS = []

  (1..N-1).each do |machine_id|
    ANSIBLE_RAW_SSH_ARGS << "-o IdentityFile=.vagrant/machines/machine#{machine_id}/#{VAGRANT_VM_PROVIDER}/private_key"
  end

  (1..N).each do |machine_id|
    config.vm.define "machine#{machine_id}" do |machine|
      machine.vm.box = "ubuntu/trusty64"
      machine.vm.hostname = "machine#{machine_id}"
      machine.vm.network "private_network", ip: "192.168.77.#{10+machine_id-1}"

      # Only execute once the Ansible provisioner,
      # when all the machines are up and ready.
      if machine_id == N
        machine.vm.provision :ansible do |ansible|
          # Disable default limit to connect to all the machines
          ansible.limit = "all"
          ansible.playbook = "main.yml"
          ansible.inventory_path = "staging"
          ansible.verbose = "-v"
          ansible.raw_ssh_args = ANSIBLE_RAW_SSH_ARGS
        end
      end
    end
  end
end