chef-client connection errors after executing knife bootstrap

Question

I'm getting these errors [http://pastebin.com/BEgNh2mm][chef-client/server errors] after successfully bootstrapping a node with chef.

The initial error which occurs is as follows

(snippet from sever logs):

merb : chef-server (api) : worker (port 4000) ~ Started request handling: Wed Aug 08 19:04:32 +0200 2012 merb : chef-server (api) :worker (port 4000) ~ Params: {"name"=>"devhouse", "controller"=>"clients", "action"=>"create", "admin"=>false} merb : chef-server (api) : worker (port 4000) ~ Connection reset by peer - (Errno::ECONNRESET)

(snippet from client STDOUT)

[2012-08-08T19:47:57+02:00] INFO: * Chef 10.12.0 * [2012-08-08T19:47:57+02:00] INFO: Client key /etc/chef/client.pem is not present - registering [2012-08-08T19:47:58+02:00] INFO: HTTP Request Returned 409 Conflict: Client already exists [2012-08-08T19:47:58+02:00] INFO: HTTP Request Returned 403 Forbidden: You are not allowed to take this action. [2012-08-08T19:47:58+02:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out [2012-08-08T19:47:58+02:00] FATAL: Net::HTTPServerException: 403 "Forbidden"

From the above the error the only thing I can deduce is that the client failed to communicate with the server, but, the mind boggling thing is that, somehow the client was able to register with the server cause executing knife client list shows client which field in the list however updated attempt by client fails with 403 "Forbidden"

Chef common errors does not seem to have any records of such behavior.

Answer 1

The problem had two parts to it:

1. Connection reset by peer - (Errno::ECONNRESET)

2. HTTP Request Returned 409 Conflict: Client already exists and FATAL: Net::HTTPServerException: 403 "Forbidden"

The first issue I was able to narrow down to chef not been able to communicate with rabbitmq, it seems somewhere along the line all settings on rabbitmq were removed.

The second issue, during my investigation (I believe this is probably documented somewhere) it became apparent that chef stores pem info in couchdb through rabbitmq before client generates it's client.pem file. Since that process was not completing due to issue 1, the client was created but authentication information was missing.

Solution

1. stop all chef services (solr, server, webui) and remove all *.pem files on chef server located under /etc/chef/
2. Ensure rabbitmq is configured with all the necessary info and update your chef server.rb file with information
3. start-up chef services (solr, server, webui), this will cause all necessary .pem files to be generated again
4. run knife client delete "node-which-had-issues"
5. on new/bootstrapped node execute chef-client