Chef-client throws 403 forbidden intermittently

Question

I am using a brand new chef-server (not enterprise), and running chef-client throws a 403 forbidden error approximately every 1 out of 5 runs. Anyone know why this might occur?

• chef-server 11.0.12 via getchef.com/chef/install
• chef-client 11.12.4 via rubygem
• ruby 2.1.2
• Debian Wheezy 7.5

The error:

198.xx.xxx.xxx ================================================================================
198.xx.xxx.xxx Error executing action `create` on resource 'template[/var/www/mysite.com/shared/config/database.yml]'
198.xx.xxx.xxx ================================================================================
198.xx.xxx.xxx
198.xx.xxx.xxx Net::HTTPServerException
198.xx.xxx.xxx ------------------------
198.xx.xxx.xxx
198.xx.xxx.xxx 403 "Forbidden"
198.xx.xxx.xxx
198.xx.xxx.xxx
198.xx.xxx.xxx Resource Declaration:
198.xx.xxx.xxx
198.xx.xxx.xxx ---------------------
198.xx.xxx.xxx # In /var/chef/cache/cookbooks/rails/recipes/production.rb
198.xx.xxx.xxx
198.xx.xxx.xxx  40: template node[:rails][:app_root]+"/shared/config/database.yml" do
198.xx.xxx.xxx  41:     owner "root"
198.xx.xxx.xxx  42:     group "xx-dev"
198.xx.xxx.xxx  43:     mode 0775
198.xx.xxx.xxx  44: end
198.xx.xxx.xxx  45:
198.xx.xxx.xxx
198.xx.xxx.xxx Compiled Resource:
198.xx.xxx.xxx
198.xx.xxx.xxx ------------------
198.xx.xxx.xxx
198.xx.xxx.xxx # Declared in /var/chef/cache/cookbooks/rails/recipes/production.rb:40:in `from_file'
198.xx.xxx.xxx
198.xx.xxx.xxx template("/var/www/mysite.com/shared/config/database.yml") do
198.xx.xxx.xxx   provider Chef::Provider::Template
198.xx.xxx.xxx   action "create"
198.xx.xxx.xxx   retries 0
198.xx.xxx.xxx   retry_delay 2
198.xx.xxx.xxx   path "/var/www/mysite.com/shared/config/database.yml"
198.xx.xxx.xxx   backup 5
198.xx.xxx.xxx   source "database.yml.erb"
198.xx.xxx.xxx   cookbook_name "rails"
198.xx.xxx.xxx   recipe_name "production"
198.xx.xxx.xxx   mode 509
198.xx.xxx.xxx   owner "root"
198.xx.xxx.xxx   group "xx-dev"
198.xx.xxx.xxx end
198.xx.xxx.xxx
198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] ERROR: Running exception handlers
198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json
198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] ERROR: Exception handlers complete
198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
198.xx.xxx.xxx [2014-05-29T20:16:34+00:00] FATAL: Net::HTTPServerException: template[/var/www/mysite.com/shared/config/database.yml] (rails::production line 40) had an error: Net::HTTPServerException: 403 "Forbidden"

Answer 1

Objects from cookbooks (templates, cookbook_files) are lazily-loaded by default.

If it's a very long run, and it sounds like it is, then the runtime might exceed the maximum time window enforced by the server from the original authentication.

You can turn off lazy loading on the client by adding

no_lazy_load true

to your client.rb. See http://docs.opscode.com/config_rb_client.html