Deployment using CodeDeploy failing at "install" step due to error "File already exists"

Question

This is the error I am getting at the "install" step -

File already exists at location /var/cake_1.2.0.6311-beta/app/webroot/../../somefile.php

I am unable to conclude what course of events brought to this scenario. But what I remember is this -

• I initiated a deployment via jenkins (code deployment plugin integrated). The deployment failed at the "install" step, with the error I pasted above.

  Note - This is the first file in the files section of the appspec.yml file.

• As far as I remember, I did some deployment on the same deployment group, directly via code deploy (most probably using the same S3 file that Jenkins created in the above step) and it had succeeded.

  • I then had several failed deployments (deliberately failed at various steps, e.g. beforeInstall by exiting with a non zero number).

Thereafter, I am not able to get any successful deployment even directly via code deploy.

Background

I checked this question which appears related and tried by deleting everything inside /opt/codedeploy-agent/deployment-root/​, as in mentioned in one of the answers, however, it resulted in corrupted code deploy in my instances and deploying via Jenkins started throwing this error saying appspec file from archive (it was trying to look for the last successful deployment, it seems, which got deleted) was not found -

Jenkins triggered code deploy is failing at ApplicationStop step even though same deployment group via code deploy directly is running successfully

I then reinstalled code deploy on the instances and the steps mentioned above are what I did after that.

Update - Opening Bounty

As answered by Rodrigo M, deleting such files from the deployment path on the instances has worked for me so far, but the same is not working for this one particular file -

 File already exists at location /var/cake_1.2.0.6311-beta/deployment/serverLoad.json

I verified the following -

• After deleting the file from the above mentioned location on all the instances, I did a new deployment. I observed that the file reappeared but still the same error as mentioned above was thrown and the deployment failed.

• Without deleting, I chmod'ed the file to 777. Same result.

Interestingly, I got this error for multiple files yesterday, and I proceeded by deleting them one by one as I found them and took a new build. The problem got fixed for all the other files except for this one file, for which even deleting is not working. I have no clue!

Note - All the deployments I am doing now are triggered by Jenkins (no direct deployments triggered from AWS code deploy).

Here is the log trail from /var/log/aws/codedeploy-agent/codedeploy-agent.log containing the relevant error stack strace -

2016-11-10 07:38:12 INFO  [codedeploy-agent(16889)]: Version file found in /opt/codedeploy-agent/.version.
2016-11-10 07:38:12 INFO  [codedeploy-agent(16889)]: [Aws::CodeDeployCommand::Client 200 0.025545 0 retries] put_host_command_complete(command_status:"Failed",diagnostics:{format:"JSON",payload:"{\"error_code\":5,\"script_name\":\"\",\"message\":\"File already exists at location /var/cake_1.2.0.6311-beta/deployment/serverLoad.json\",\"log\":\"\"}"},host_command_identifier:"WyJjb20uYW1hem9uLmFwb2xsby5kZXBsb3ljb250cm9sLmRvbWFpbi5Ib3N0Q29tbWFuZElkZW50aWZpZXIiLHsiZGVwbG95bWVudElkIjoiQ29kZURlcGxveS91cy1lYXN0LTEvUHJvZC9hcm46YXdzOnNkczp1cy1lYXN0LTE6Mzc3NzAzOTYxOTk4OmRlcGxveW1lbnQvZC1VOVFPR0RBWUkiLCJob3N0SWQiOiJhcm46YXdzOmVjMjp1cy1lYXN0LTE6Mzc3NzAzOTYxOTk4Omluc3RhbmNlL2ktZWNmYzU1YTkiLCJjb21tYW5kTmFtZSI6Ikluc3RhbGwiLCJjb21tYW5kUG9zaXRpb24iOjQsImNvbW1hbmRBdHRlbXB0IjoxfV0=")

2016-11-10 07:38:12 ERROR [codedeploy-agent(16889)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error during perform: RuntimeError - File already exists at location /var/cake_1.2.0.6311-beta/deployment/serverLoad.json - /opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:115:in `generate_normal_copy'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:67:in `block (2 levels) in generate_instructions'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:55:in `each'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:55:in `block in generate_instructions'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/install_instruction.rb:68:in `generate_instructions'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:54:in `generate_instructions'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/installer.rb:34:in `install'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:114:in `block in <class:CommandExecutor>'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_executor.rb:62:in `execute_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:132:in `process_command'
/opt/codedeploy-agent/lib/instance_agent/plugins/codedeploy/command_poller.rb:65:in `perform'
/opt/codedeploy-agent/lib/instance_agent/agent/base.rb:28:in `run'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:38:in `block in run'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:55:in `with_error_handling'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:37:in `run'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:70:in `block in run_with_error_handling'
/opt/codedeploy-agent/lib/instance_agent/runner/child.rb:55:in `with_error_handling'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:69:in `run_with_error_handling'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:33:in `block in start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:22:in `loop'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/child.rb:22:in `start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:206:in `block in spawn_child'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in `fork'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:204:in `spawn_child'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:196:in `block in spawn_children'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:195:in `times'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:195:in `spawn_children'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:134:in `start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:37:in `block in start'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:36:in `fork'
/opt/codedeploy-agent/vendor/gems/process_manager-0.0.13/lib/process_manager/master.rb:36:in `start'
/opt/codedeploy-agent/bin/../lib/codedeploy-agent.rb:41:in `block (2 levels) in <main>'
/opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/command_support.rb:130:in `call'
/opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/command_support.rb:130:in `execute'
/opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/app_support.rb:262:in `block in call_command'
/opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/app_support.rb:275:in `call'
/opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/app_support.rb:275:in `call_command'
/opt/codedeploy-agent/vendor/gems/gli-2.5.6/lib/gli/app_support.rb:69:in `run'
/opt/codedeploy-agent/bin/../lib/codedeploy-agent.rb:88:in `<main>'

Update 2

I was able to apply a dirty fix, by renaming the json file and removing the old file reference in appspec.yml. However, on a new deployment group (with new ec2 instances) the new json file is causing the same file exits issue. It is painful to change the file name every time. Very irritated by what is happening.

Answer 1

As part of its processes, CodeDeploy will look for information on previously deployed files for the app/deployment group. If then it uses this info to delete the existing file(s), preparing for the deployment of the new revisions, as needed.

http://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-steps.html#deployment-rollback

In this case, there is some inconsistent reference due to manual cleanup you may have done previously.

A good option for all deployments is to simply delete all files in the deployment folder during the BeforeInstall hook. This will fix this issue now and going forward.