How to access buckets with boto3

Question

Here are my permissions:

Additionally, I have this as a bucket policy:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::manga-learn-data",
                "arn:aws:s3:::manga-learn-data/*"
            ]
        }
    ]
}

I have this in my ~/.aws/config file:

[default]
region=us-west-2

And this in my ~/.aws/credentials file:

[default]
aws_access_key_id = <access-key>
aws_secret_access_key = <secret-key>

Now I do:

>>> import boto3
>>> s3 = boto3.resource('s3')
>>> s3.buckets.all()
s3.bucketsCollection(s3.ServiceResource(), s3.Bucket)
>>> for bucket in s3.buckets.all():
...         print(bucket.name)
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/alex/anaconda2/lib/python2.7/site-packages/boto3/resources/collection.py", line 83, in __iter__
    for page in self.pages():
  File "/Users/alex/anaconda2/lib/python2.7/site-packages/boto3/resources/collection.py", line 161, in pages
    pages = [getattr(client, self._py_operation_name)(**params)]
  File "/Users/alex/anaconda2/lib/python2.7/site-packages/botocore/client.py", line 262, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/alex/anaconda2/lib/python2.7/site-packages/botocore/client.py", line 552, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied

And you see the traceback there. I am following the steps here: https://github.com/boto/boto3

Any suggestions?

Answer 1

Your code currently tries to list all buckets but the IAM user does not have permission to do that.

You either have to grant the ListAllMyBuckets access to your IAM user, e.g.:

    {
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "arn:aws:s3:::*"
    },

Or you need to change your code to only access the bucket you are interested in:

bucket = s3.Bucket('manga-learn-data')
for object in bucket:
    # do whatever you need to do here