Best common workflow for Vagrant and Knife Solo (or any other non Vagrant Chef Env)?

Question

After a while away from Chef I am currently getting back to it for all my server needs. I already used it a few years back for a while and got to like it, but had other people do the ops part lately.

But I am still struggling to find a really good starting setup (local Vagrant+Chef) to go with for development. Using Vagrant by itself is a breeze, super easy to use and reasonably quick to develop the server setup on top of it. But once I have to leave Vagrant for the actual deployment it gets annoying again. I cant always use a Vagrant box, but Rackspace or even an actual hardware box somewhere sometimes.

Right now I do it as follows:

Develop against Vagrant, trying to have as few actual config in the Vagrantfile as possible

• using mostly roles, with pre-defined default attributes for my usual environments
• only specifying node specific attributes in the Vagrantfile
• running the vagrant provisioning with manually snapshots directly from VirtualBox in between steps to avoid re-building from scratch all the time

Stage and actually deploy the server:

Using Rackspace Cloud (staging at least) with Knife Solo

• creating Rackspace Servers from the Web-Interface to have consistent first provisioning step via (roo@ip with password) for all servers
• add public-key auth to the server so the pw is not needed anymore
• add a local node configuration for the new server copying what is defined in the Vagrantfile
• use knife solo prepare+bootstrap the server

This is already not so bad but still has a few glitches that are sometimes quite annoying:

• manual snapshots from VirtualBox for Vagrant (i did not find any working plugin yet)
• redundancy between node config and Vagrantfile (probably have the Vagrantfile parse a node config and use the values that way)
• inconsistent bootstrapping between Vagrant and Knife Solo (probably using a common script for both if I figure out how to add it to Vagrant (*1))

In addition to that I had to sadly find out that the chef-recipes have widely grown into quite a jungle of incompatible and opinionated artefacts. Its sometimes quite hard to even get a basic setup with the default recipes working. I am a little surprised that a lot of the basics are hardly covered:

Getting the combination of sshd+iptables to work took me a day of research and then still modifying the default templates to get it to work - while I expected it to be the starting point for almost any server. Also there does not seem to be any default chef-user workflow. Everything I found so far either runs as root or needs quite a bit modification. And last but not least chef (on ubuntu 12.04) still uses ruby 1.8.7 which is reaching its end of live in just a few months.

It might be that I just did not find the right ressources to cover all the points I am currently still struggling with or satisfied with but still it seems there are quite a lot ways to improve it.

So how does vagrant + chef work in a real environment (beyond just the local virtual box) work for you and what pitfalls are there to watch out for?

It seems that it starts insanely awesome having all that automated bootstrapping through vagrant locally but once you go beyond that things get really messy. I'd be glad if people also using some kind of setup like this could give me some pointers of how to tackle the above mentioned issues. I don't mind putting in some effort to get it all running as i expect it to be but maybe i already walked down the wrong part and just make it all harder on myself that in actually needs to be ;)

For now the short summary is: Vagrand+ChefSolo (KnifeSolo) is quite awesome, but to work properly the whole bootstrapping part needs to we switched out with a custom one to get a proper System base before applying cookbooks - and those needs to be carefully picked out of the jungle.

Progess/Update Notes

(*1): Just figured out, purely by luck when giving it a shot, that apparently it is possible to add more than one provision-mechanism in one Vagrantfile:

config.vm.provision :shell, path: 'bootstrap.ubuntu-12.04.2-server-amd64.sh'
  config.vm.provision :chef_solo do |chef|
    ...
  end

Since the shell one is executed first I can use it as a custom preperation for Vagrant servers while still using Chef-Solo for the actual Setup. Yay. Still will have to see how useful this will be at the end but a bit missing step works now to have the processes aligned.

Answer 1

You can even do more than one instance of a provision mechanism.

I run a squid proxy on my laptop, so my standard Vagrantfile includes one shell provider that sets up a global http proxy in /etc/profile.d/proxy.sh that points at my squid, and another shell provider that copies a CentOS repo file into the vm that uses a specific mirror rather than using the fastest mirror so that I minimize cache misses. After those run, there is a chef provider in the Vagrantfile that actually configures the VM.