Why can't my puppet agent on a vagrant vm connect to my puppetmaster service on its vm?

Question

My goal is to have one or two nodes, and one puppetmaster. I used a bash script to provision each node with the puppetlabs repository and install the latest version of puppet and/or puppetmaster. But every time I try to run puppet agent --test on the node, it returns this error:

root@vm:~# puppet agent --test
Error: Could not request certificate: Connection refused - connect(2)
Exiting; failed to retrieve certificate and waitforcert is disabled

• I've confirmed that the puppetmaster service is running.
• I don't turn on the node until the master is up.
• Running puppet cert list shows no certs waiting to be approved on the puppetmaster.
• My /etc/hosts files have the ip addresses and hostnames configured correctly.
• /etc/puppet/puppet.conf looks right.
• I can ping the node from the master, and the other way around.
• iptables --list shows no firewall rules.

Is it that Vagrant/Virtualbox can't figure out how to deal with port 8140? The error says 'connection refused', so I thought it was a firewall issue. But there's no firewall...

So, where did I screw up?

Here's my Vagrantfile:

Vagrant.configure("2") do |config|

  config.vm.define :puppetmaster do |puppetmaster|
    puppetmaster.vm.box = "ubuntu-server-12042-x64-vbox4210-nocm"
    puppetmaster.vm.box_url = "http://puppet-vagrant-boxes.puppetlabs.com/ubuntu-server-12042-x64-vbox4210-nocm.box"
    puppetmaster.vm.network :private_network, ip: "192.168.77.1"
    #puppetmaster.vm.network :forwarded_port, guest: 80, host: 20001
    #puppetmaster.vm.network :forwarded_port, guest: 443, host: 24431
    #puppetmaster.vm.network :forwarded_port, guest: 22, host: 20022
    puppetmaster.vm.hostname = "vm.puppetmaster.lab"
    puppetmaster.vm.provision :shell, :path => "master-bootstrap.sh"
    puppetmaster.vm.synced_folder "modules/", "/etc/puppet/modules"
    puppetmaster.vm.synced_folder "manifests/", "/etc/puppet/manifests"
  end

  config.vm.define :alpha do |alpha|
    alpha.vm.box = "ubuntu-server-12042-x64-vbox4210-nocm"
    alpha.vm.box_url = "http://puppet-vagrant-boxes.puppetlabs.com/ubuntu-server-12042-x64-vbox4210-nocm.box"
    alpha.vm.network :private_network, ip: "192.168.77.2"
    #alpha.vm.network :forwarded_port, guest: 22, host: 20023
    alpha.vm.hostname = "vm.alpha.lab"
    alpha.vm.provision :shell, :path => "alpha-bootstrap.sh"
  end


  config.vm.define :beta do |beta|
    beta.vm.box = "ubuntu-server-12042-x64-vbox4210-nocm"
    beta.vm.box_url = "http://puppet-vagrant-boxes.puppetlabs.com/ubuntu-server-12042-x64-vbox4210-nocm.box"
    beta.vm.network :private_network, ip: "192.168.77.3"
    #beta.vm.network :forwarded_port, guest: 22, host: 20024
    beta.vm.hostname = "vm.beta.lab"
    beta.vm.provision :shell, :path => "beta-bootstrap.sh"
  end
end

My node bootstrap bash script:

#!/usr/bin/env bash

wget http://apt.puppetlabs.com/puppetlabs-release-precise.deb
dpkg -i puppetlabs-release-precise.deb
apt-get update
#apt-get -y dist-upgrade
apt-get -y install puppet

echo '192.168.77.1 vm.puppetmaster.lab' >> /etc/hosts

echo '[agent]' >> /etc/puppet/puppet.conf
echo 'server=vm.puppetmaster.lab' >> /etc/puppet/puppet.conf
echo 'certname=vm.alpha.lab' >> /etc/puppet/puppet.conf

My master bootstrap bash script:

#!/usr/bin/env bash

wget http://apt.puppetlabs.com/puppetlabs-release-precise.deb
dpkg -i puppetlabs-release-precise.deb
apt-get update
#apt-get -y dist-upgrade
apt-get -y install puppet
puppet apply /etc/puppet/manifests/default.pp

Note that I'm using a slightly modified puppet module from Pro Puppet to configure/install puppet/puppetmaster. That's why I run the puppet apply command in the master bootstrap script.

Edit I can get the two vm's communicating if I use :public_network and let my workplaces dhcp server assign the ip addresses. 'Course, this is not ideal since that means I can't just use a 192.x.x.x address on a private, local virtual network. I have to configure the hosts files manually before running any puppet stuff. But at least I know the issue is something to do with using :private_network.

Edit 2 I just tried using the puppetlabs/firewall module to force 8140 open on the puppetmaster vm's firewall, just in case there was something I was missing when I checked the firewall before. That let me run the puppet agent on that vm. But when I tried on the alpha vm, it had the same connection refused error. So, I doubt the firewall has anything to do with it.

Edit 3 The private network option sets up a second NIC on the VM. The first NIC is NAT'd, the second is what gets the static IP I assign.

Puppet is listening on both NIC's. (At least as far as I can tell.)

The 192.x.x.x ip addresses are unique to these VM's. That range is not used anywhere else.

Answer 1

Lookup ifconfig in your host. May be ip 192.168.77.1 is address of your host for private networking with VM. Change ip of puppetmaster VM, destroy it and up it.

Answer 2

I'm brand new to puppet myself. I just ran across this a few hours ago.

The connection refused portion means that the agent can't find your puppet master. This means one of a few things:

1. The server it is connecting to isn't correct. You can use --server whatever.your.server.is to force it if you think this is the problem.
2. Puppet master isn't running on the server. If you think this is the problem, you can try running netstat -an | grep "LISTEN" on the server and see if you see port 8140 running something.
3. There is some connectivity issue between the box, like perhaps a firewall. Try pinging the box, or try telnetting to that port with "telnet whatever.your.server.is 8140". If it connects that isn't the problem.

One of those three are probably it.