Amazon Beanstalk Errors on Deployment

Question

I am running two environments and I am getting the same errors on each upon deployment of an application. These errors also happen on deployment of the Beanstalk Sample Application. The errors go away when we create a new environment and deploy the same files in it.. for a few days at least and then they will return.

Instance: i-72715b7f Command failed on instance. An unexpected error has occurred ErrorCode: 0000000001.

Update environment operation is complete, but with errors. For more information, see troubleshooting documentation.

Instance: i-85437188 Module: AWSEBAutoScalingGroup ConfigSet: null Command failed on instance. Return code: 1 Output: Error occurred during build: Command hooks failed .

Script /opt/elasticbeanstalk/hooks/appdeploy/enact/99_reload_app_server.sh failed with returncode 1

Script /opt/elasticbeanstalk/hooks/appdeploy/pre/12_update_permissions.sh failed with returncode 1

The application (WordPress website) seems to work fine even with these errors in the events, except for with random problems that act like traditional permission errors (cannot upload images, or permalinks not working). We had been running the same application on beanstalk with no problems or errors at all. These errors also happen on deployment of the Beanstalk Sample Application.

This is the only snippet from the error log that seems to have any relevance.

2014-09-17 19:47:08,825 ERROR Error encountered during build of Hook-EnactAppDeploy: Command hooks failed
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build
changes = CommandTool().apply(self._config.commands)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply
raise ToolError(u"Command %s failed" % name)
ToolError: Command hooks failed
2014-09-17 19:47:08,826 ERROR Unhandled exception during build: Command hooks failed
Traceback (most recent call last):
File "/opt/aws/bin/cfn-init", line 122, in <module>
worklog.build(detail.metadata, configSets)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 117, in build
Contractor(metadata).build(configSets, self)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 502, in build
self.run_config(config, worklog)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 511, in run_config
CloudFormationCarpenter(config, self._auth_config).build(worklog)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/construction.py", line 247, in build
changes = CommandTool().apply(self._config.commands)
File "/usr/lib/python2.6/site-packages/cfnbootstrap/command_tool.py", line 113, in apply
raise ToolError(u"Command %s failed" % name)
ToolError: Command hooks failed

What is wrong here? How do we fix it?

Answer 1

May be too late, but recording for posterity: whenever the root file system is full on the EB instance, this error appears. There may be other reasons for this error, but this sure is one.

In our case it was a chatty logging setting that escaped dev environment in a build and got deployed, clogging up /var/log. A quick eb health revealed a warning of the form Degraded 100 % of root file system is in use. 0 MB free. From there, it was easy to clean up.

Answer 2

The problem ended up being with a faulty WordPress plugin. The plugin was creating a new folder in the /tmp directory every time that a file was uploaded but it was never deleting the folder. It would delete all the files within the folder, but never the folder. As the clients used the site and continued to upload images, then those folders would add up and it would eventually fill up the /tmp directory. Fixing this plugin solved the issue and it has not returned.