How do I deploy a Python application to Amazon Elastic Beanstalk from Jenkins?

Question

I am trying to deploy to Amazon Elastic Beanstalk programmatically from a Jenkins job. On my development machine, this is as simple as:

eb deploy $(AWS_ELASTIC_BEANSTALK_ENVIRONMENT)

On Jenkins, it should be as simple as configuring the following as a build command:

virtualenv env && source env/bin/activate && pip install awsebcli
mkdir -p .elasticbeanstalk
cat << EOF > .elasticbeanstalk/config.yml 
branch-defaults:
  master:
    environment:  myenv
global:
  application_name:  myapp
  default_ec2_keyname: null
  default_platform: 64bit Amazon Linux 2014.09 v1.0.9 running Python 2.7
  default_region: us-west-2
  profile: eb-cli
  sc: git
EOF
eb deploy myenv

However, this fails with the following trace:

[EnvInject] - Loading node environment variables.
Building remotely on standard Amazon Linux 2014.09 AMI (i-d39710df) (x) in workspace /media/ephemeral0/jenkins/workspace/My_Job
Fetching changes from the remote Git repository
Fetching upstream changes from [email protected]:myapp.git
Checking out Revision be45db94111f9ab49fe8031eb583307d2fb9921c (origin/master)
[My_Job] $ /bin/sh -xe /tmp/hudson8633484437962332339.sh
+ virtualenv env
New python executable in env/bin/python2.7
Not overwriting existing python script env/bin/python (you must use env/bin/python2.7)
Installing Setuptools..................................................................................................................done.
Installing Pip....................................................................................................................................done.
+ source env/bin/activate
++ deactivate nondestructive
++ unset pydoc
++ '[' -n '' ']'
++ '[' -n '' ']'
++ '[' -n /bin/sh -o -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ '[' '!' nondestructive = nondestructive ']'
++ VIRTUAL_ENV=/media/ephemeral0/jenkins/workspace/My_Job/env
++ export VIRTUAL_ENV
++ _OLD_VIRTUAL_PATH=/usr/local/bin:/bin:/usr/bin:/opt/aws/bin
++ PATH=/media/ephemeral0/jenkins/workspace/My_Job/env/bin:/usr/local/bin:/bin:/usr/bin:/opt/aws/bin
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1=
++ '[' x '!=' x ']'
+++ basename /media/ephemeral0/jenkins/workspace/My_Job/env
++ '[' env = __ ']'
+++ basename /media/ephemeral0/jenkins/workspace/My_Job/env
++ PS1='(env)'
++ export PS1
++ alias 'pydoc=python -m pydoc'
++ '[' -n /bin/sh -o -n '' ']'
++ hash -r
+ pip install awsebcli
Requirement already satisfied (use --upgrade to upgrade): awsebcli in ./env/lib/python2.7/site-packages
Downloading/unpacking setuptools>=7.0 (from awsebcli)
  Running setup.py egg_info for package setuptools

Requirement already satisfied (use --upgrade to upgrade): pyyaml>=3.11 in ./env/lib/python2.7/site-packages (from awsebcli)
Requirement already satisfied (use --upgrade to upgrade): six==1.8.0 in ./env/lib/python2.7/site-packages (from awsebcli)
Requirement already satisfied (use --upgrade to upgrade): cement==2.4 in ./env/lib/python2.7/site-packages (from awsebcli)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil>=2.2 in ./env/lib/python2.7/site-packages (from awsebcli)
Requirement already satisfied (use --upgrade to upgrade): jmespath>=0.4.1 in ./env/lib/python2.7/site-packages (from awsebcli)
Installing collected packages: setuptools
  Found existing installation: setuptools 0.9.7
    Uninstalling setuptools:
      Successfully uninstalled setuptools
  Running setup.py install for setuptools

    Installing easy_install script to /media/ephemeral0/jenkins/workspace/My_Job/env/bin
    Installing easy_install-2.7 script to /media/ephemeral0/jenkins/workspace/My_Job/env/bin
Successfully installed setuptools
Cleaning up...
+ mkdir -p .elasticbeanstalk
+ cat
+ cat .elasticbeanstalk/config.yml
branch-defaults:
  master:
    environment: myenv
global:
  application_name: myapp
  default_ec2_keyname: null
  default_platform: 64bit Amazon Linux 2014.09 v1.0.9 running Python 2.7
  default_region: us-west-2
  profile: eb-cli
  sc: git
+ eb deploy myenv
ERROR: The config profile (eb-cli) could not be found
Build step 'Execute shell' marked build as failure
Finished: FAILURE

It's unclear why this happens since when I run the above on a local copy of my project it works fine.

The error message doesn't seem to be much help. It's unclear why eb-cli would be unable to found on the Jenkins machine.

So to summarize again my question: How do I deploy to Amazon Elastic Beanstalk from Jenkins? Is the above approach correct, but with errors in the details? Or is there some simpler way entirely?

Answer 1

To correct the config profile (eb-cli) could not be found error, drop the credentials you are using to deploy to EB into ~/.aws/config for your jenkins user on your jenkins machine. If you built your deployment on a local machine, you should be able to pull the file directly from ~/.aws/config locally. It will look like this:

[profile eb-cli]
aws_access_key_id = (for your IAM user)
aws_secret_access_key = (for your IAM user)

Answer 2

I resolved this by sshing into the Jenkins machine, running eb init, and then comparing the generated .elasticbeanstalk/config.yml with the one in the here-doc I was using above. The two were different because of the different security profiles on my development machine versus the Jenkins machine.

We can rewrite this script to be more robust against different config.yaml files like this:

virtualenv env && source env/bin/activate && pip install awsebcli
echo "1" | eb init myapp --region us-west-2 && eb use myenv && eb deploy myenv

Note, we use echo "1" | eb init myapp --region us-west-2 to select a default environment since eb init does not take environment as a positional argument and then use eb use myenv to select the environment we want.