Recommended way to manage credentials with multiple AWS accounts?

Question

What is the best way to manage multiple Amazon Web Services (AWS) accounts through boto?

I am familiar with BotoConfig files, which I'm using. But each file describes only a single account...and I am working with more than just the one organization. For all the usual legal, financial, and security reasons, those accounts cannot be commingled.

Currently I am using one boto config file per account. E.g.:

• ~/.boto default account
• ~/.boto_clowncollege for "clowncollege" account
• ~/.boto_razorassoc for "razorassoc" account
• ~/.boto_xyz for "xyz" account

Then something like:

def boto_config_path(account=None):     """     Given an account name, return the path to the corresponding boto     configuration file. If no account given, return the default config file.     """     path = '~/.boto' + ('_' + account if account else '')     clean_path = os.path.abspath(os.path.expanduser(path))     if os.path.isfile(clean_path):         return clean_path     else:         errmsg = "cannot find boto config file {} for {}".format(clean_path, account)         raise ValueError(errmsg)  def aws_credentials(account=None):     """     Return a tuple of AWS credentials (access key id and secret access key) for     the given account.     """     try:         cfg = INIConfig(open(boto_config_path(account)))         return ( cfg.Credentials.aws_access_key_id, cfg.Credentials.aws_secret_access_key )     except Exception:         raise  conn = EC2Connection(*aws_credentials('razorassoc')) 

Good, bad, or indifferent? Suggested improvements?

Answer 1

updated 2015-02-06, corrected 2015-03-19 by following top section

New standardized sharing of boto and AWSCLI credentials (boto>==2.29.0)

Since boto 2.29 there is new easy way for sharing BOTO and AWS CLI credentials as described by Mike Garnaat in A New and Standardized Way to Manage Credentials in the AWS SDKs

The aim is to:

1. allow sharing credentials by boto, AWSCLI and possibly other SDKs
2. keep all configuration in single file, located in user profile directory
3. allow using named profiles
4. keep it as simple as possible (e.g. prevent conflicts with other methods)

Create credentials file

Create the file ~/.aws/credentials (Mac/Linux) or %USERPROFILE%\.aws\credentials (Windwos) as follows:

[default] aws_access_key_id = AxxxA aws_secret_access_key = Zxxxr region = eu-west-1  [jekyl] aws_access_key_id = AxxxA aws_secret_access_key = Zxxxr region = eu-west-1  [hyde] aws_access_key_id = AxxxZ aws_secret_access_key = CxxxZ region = eu-west-1 

From now on, you may use a code like this:

Use default profile

import boto con = boto.connect_s3() 

Use explicit profile set by AWS_PROFILE env. var

(this is my favourite option keeping profile name out of code and still giving the deployer of my application a chance to pick specific profile)

$ export AWS_PROFILE=jekyl 

and keep your code as simple as before:

import boto con = boto.connect_s3() 

Specify explicit profile in your code

import boto con = boto.connect_s3(profile_name="jekyl") 

This is all you typically need to do

The logic for picking proper credentials is described in boto issue #2292 as follows:

The loading order from highest to lowest precedence:

1.Directly passed from code

2. Environment variables for key/secret

3. Environment variables for profile

4. Shared credential file explicit profile

5. Shared credential file default profile

6. Config file explicit profile

7. Config file Credentials section

A profile passed from code overrides any set in an environment variable.

To keep things clean and simple, it is good to get rid of older methods, so remove any old style files (like ~/.aws/config or ~/.boto), unset environmental varialbe BOTO_CONFIG if set and possibly also the file, to which such variable points to.

And that is really all for boto >=2.29.0

Note: Do not attempt to control location of config file by env.variable (like AWS_CONFIG_FILE), it does not work as expected.

Use boto config profile (boto>=2.24.0)

Following description is kept here only for those, who cannot upgrade to boto 2.29.0 or higher

Since boto 2.24.0 there is a feature called profile_name

In your ~/.boto file you already have your [Credentials] section, this will serve as fallback option, and then [profile ] sections serving for different profiles:

[Credentials] aws_access_key_id = AxxxA aws_secret_access_key = Zxxxr  [profile jekyl] aws_access_key_id = AxxxA aws_secret_access_key = Zxxxr  [profile hyde] aws_access_key_id = AxxxZ aws_secret_access_key = CxxxZ 

Then, when creating connection, you use this way:

import boto con = boto.connect_s3(profile_name="jekyl") 

Note, that this feature is available since boto 2.24.0.

Tutorial is here http://docs.pythonboto.org/en/latest/boto_config_tut.html?highlight=profile

There are even some notes about using keyrings, but I will first get used to this profile stuff, which I was dreaming of few years.

Sharing config file with AWSCLI

AWSCLI became really great tool. As format of config file is almost the same, I use it in following way:

1. keep ~/.aws/config file as created by AWSCLI (this is default location)
2. copy section [default] and rename it to [Credentials] (leaving the same values inside).
3. add whatever profiles I use
4. set BOTO_CONFIG variable to point to this ~/.aws/config file.

The ~/.boto would then become `~/.aws/config with following content:

[default] aws_access_key_id = AxxxA aws_secret_access_key = Zxxxr  [Credentials] aws_access_key_id = AxxxA aws_secret_access_key = Zxxxr  [profile jekyl] aws_access_key_id = AxxxA aws_secret_access_key = Zxxxr  [profile hyde] aws_access_key_id = AxxxZ aws_secret_access_key = CxxxZ 

This way, it gets shared for both AWSCLI and boto including profiles.

Answer 2

In the future, boto will provide better tools to help you manage multiple credentials but at the moment, there are a couple of environment variables that might help out.

First, you can set BOTO_CONFIG to point to a boto config file that you want to use and it will override any config file found in the normal locations.

Secondly, you can set BOTO_PATH to a colon-separated list of places to look for a boto config file and it will search there first, prior to the normal search locations.

Neither of those give you exactly what you want but it may make it easier to accomplish with a bit less code.

If you have ideas about how you would like this to work in boto, please let me know!