Boto3 error: The AWS Access Key Id you provided does not exist in our records

Question

I am currently trying to get access to Amazon S3 inside a virtual machine and download files like so:

s3 = boto3.resource('s3',
         aws_access_key_id="xxxxxxxxxxx",
         aws_secret_access_key="xxxxxxxxxxxxxxxxx")
s3client = boto3.client('s3')

bucket = s3.Bucket('bucketone')

for obj in bucket.objects.all():
    s3client.download_file(bucket_name, obj.key, filename)

But I’m getting the error:

botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the ListObjects operation: The AWS Access Key Id you provided does not exist in our records.

What could I be doing wrong? I checked my aws_access_key_id and aws_secret_access_key multiple times, but still getting the same error. The same code locally, but not on a virtual machine, actually works on a different computer as well. There is a reason why I’m hardcoding in the keys, as I have to.

Answer 1

You need to set the access for the boto3 session. You don't really want to put your keys in your code. What I would recommend doing first is running 'aws configure' and setting your aws_access_key_id and aws_secret_access_key in your .credentials file. Then in your code do the following:

session = boto3.Session(profile_name='name_of_your_profile')

If you have just the default profile, you might not need to do that or for good measure, just put:

session = boto3.Session(profile_name='default')

Once you have that in your code you can establish a connection to s3 with:

s3 = session.resource('s3')
bucket = s3.Bucket('bucketone')
for obj in bucket.objects.all():
   print(obj.key)

There is some problem with your code as well. You are creating an s3 client. S3 client does not have a Bucket method or property. To do the same thing with the s3 client you would do:

s3client = session.client('s3')
response = s3client.get_object(Bucket='bucketone', key='your key')

You can then iterate through the response that is returned to see the list of objects in the bucket.

That should take care of your error.

Answer 2

Boto3 users BEWARE

TL;DR

If you are using temporary credentials to connect to AWS services through Boto3, you MUST include a current aws_session_token as a parameter to your boto3.session.Session instance.

from boto3.session import Session

# Ideally this is picked up your ENV.
id_ = "<id>"
secret = "<secret>"
token = "token"

session = Session(
                aws_access_key_id=id_,
                aws_secret_access_key=secret,
                aws_session_token=token,
                region_name='<region>'
            )

# Test it on a service (yours may be different)
s3 = session.resource('s3')

# Print out bucket names
for bucket in s3.buckets.all():
    print(bucket.name)

Explanation

This is a crucial piece of information when you are testing credentials in Boto3: The error you receive may say this,

ClientError: An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.

but may mean you are missing an aws_session_token if you are using temporary credentials (in my case, role-based credentials).

According to AWS documentation, these are the parameters available to a boto3.session.Session object, however, there is no indication or clarification when it comes to this behavior in Boto3:

Parameters
aws_access_key_id (string) -- AWS access key ID
aws_secret_access_key (string) -- AWS secret access key
aws_session_token (string) -- AWS temporary session token
region_name (string) -- Default region when creating new connections
botocore_session (botocore.session.Session) -- Use this Botocore session instead of creating a new default one.
profile_name (string) -- The name of a profile to use. If not given, then the default profile is used.

Regarding the aws_session_token

Specifies an AWS session token used as part of the credentials to authenticate the user. A session token is required only if you manually specify temporary security credentials.

Resources

• aws_session_token
• Common scenarios for roles: Users, applications, and services
• Boto3 Credentials
• Session Reference