Chef: Why are resources in an "include_recipe" step being skipped?

Question

Chef seems to be processing resources in a strange order, causing my build to fail. My main recipe (mytardis-chef/site-cookbooks/recipes/default.rb) starts like this:

include_recipe "build-essential" include_recipe "mytardis::deps" include_recipe "mytardis::nginx" include_recipe "mytardis::postgresql" 

mytardis-chef/cookbooks/build-essential/recipes/default.rb looks like this:

case node['platform'] when "ubuntu","debian"   %w{build-essential binutils-doc}.each do |pkg|     package pkg do       action :install     end   end when "centos","redhat","fedora","scientific"   %w{gcc gcc-c++ kernel-devel make}.each do |pkg|     package pkg do       action :install     end   end end 

... (it's an older version of https://github.com/opscode-cookbooks/build-essential/blob/master/recipes/default.rb)

At runtime, for reasons I don't understand, this build-essential recipe gets loaded but not executed:

[default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant [default] -- v-csr-3: /tmp/vagrant-chef-1/chef-solo-3/roles [default] -- v-csc-2: /tmp/vagrant-chef-1/chef-solo-2/cookbooks [default] -- v-csc-1: /tmp/vagrant-chef-1/chef-solo-1/cookbooks [default] -- v-csdb-4: /tmp/vagrant-chef-1/chef-solo-4/data_bags [default] Running provisioner: Vagrant::Provisioners::ChefSolo... [default] Generating chef JSON and uploading... [default] Running chef-solo... [Sun, 08 Jul 2012 05:14:32 +0200] INFO: *** Chef 10.12.0 *** [Sun, 08 Jul 2012 05:14:32 +0200] DEBUG: Building node object for lucid32 [Sun, 08 Jul 2012 05:14:32 +0200] DEBUG: Extracting run list from JSON attributes provided on command line [Sun, 08 Jul 2012 05:14:32 +0200] INFO: Setting the run_list to ["recipe[mytardis]"] from JSON [Sun, 08 Jul 2012 05:14:32 +0200] DEBUG: Applying attributes from json file [Sun, 08 Jul 2012 05:14:32 +0200] DEBUG: Platform is ubuntu version 10.04 [Sun, 08 Jul 2012 05:14:32 +0200] INFO: Run List is [recipe[mytardis]] [Sun, 08 Jul 2012 05:14:32 +0200] INFO: Run List expands to [mytardis] [Sun, 08 Jul 2012 05:14:32 +0200] INFO: Starting Chef Run for lucid32 [Sun, 08 Jul 2012 05:14:32 +0200] INFO: Running start handlers [Sun, 08 Jul 2012 05:14:32 +0200] INFO: Start handlers complete. ... [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe mytardis via include_recipe [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe default in cookbook mytardis [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe build-essential via include_recipe [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe default in cookbook build-essential [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe mytardis::deps via include_recipe [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe deps in cookbook mytardis [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe mytardis::nginx via include_recipe [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe nginx in cookbook mytardis [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe iptables via include_recipe [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe default in cookbook iptables [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe mytardis::postgresql via include_recipe [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe postgresql in cookbook mytardis [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe postgresql::server via include_recipe [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe server in cookbook postgresql [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Loading Recipe postgresql::client via include_recipe [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: Found recipe client in cookbook postgresql [Sun, 08 Jul 2012 05:14:33 +0200] INFO: Processing package[postgresql-client] action install (postgresql::client line 37) [Sun, 08 Jul 2012 05:14:33 +0200] DEBUG: package[postgresql-client] checking package status for postgresql-client .... [Sun, 08 Jul 2012 05:14:45 +0200] ERROR: gem_package[pg] (postgresql::client line 42) has had an error . make sh: make: not found 

That is, the build-essential recipe is "found" and "loaded", but it's the postgres recipe that gets "processed" first. And since build-essential (which installs the C compiler) wasn't run, the latter fails.

The relevant part of my Vagrantfile looks like this:

  config.vm.provision :chef_solo do |chef|      chef.log_level = :debug      chef.cookbooks_path = ["mytardis-chef/site-cookbooks", "mytardis-chef/cookbooks"]      chef.roles_path = "mytardis-chef/roles"      chef.data_bags_path = "mytardis-chef/data_bags"      chef.add_recipe "mytardis"       end 

I was previously using a slightly earlier version of Chef (perhaps 10.10.0?) In that version, build-essential was also not being run, but mytardis::deps was. Now using Chef 10.12.0. Physical machine is OSX, VM is Ubuntu Lucid.

So, a few questions:

1. Why is build-essential not being "processed"?
2. What's the right way to do this? (I didn't write these recipes, and I understand they do or have worked for their author.)
3. Should the site-cookbooks and cookbooks 'shadowing' feature still work? It's supposedly deprecated: http://tickets.opscode.com/browse/CHEF-2308 (I tried making a symbolic link under from site-cookbooks/mytardis/recipes/build-essential but no joy).

Answer 1

It turns out this is a pretty normal (but under-documented) part of how Chef works: it compiles everything, and then starts running. Except some recipes (like Postgres) jump the queue to actually install at compile-time, using code like this:

  action :nothing end.run_action(:run) 

The solution is that anything that needs to run before Postgres also needs to do this. Fortunately, newer versions of Build-essentials allow this. You set attributes on a role as follows:

name "myapp" run_list(   "recipe[build-essential]",   "recipe[myapp]" ) default_attributes(   "build_essential" => {     "compiletime" => true   } ) 

Answer 2

if writing a cookbook, add the attribute to your attributes file:

node.default['build_essential']['compiletime'] = true 

Thanks to Colin above.