Get all s3 buckets given a prefix

Question

Currently we have multiple buckets with an application prefix and a region suffix e.g. Bucket names

• myapp-us-east-1
• myapp-us-west-1

Is there a way of finding all buckets given a certain prefix? Is there something like:

s3 = boto3.resource('s3')
buckets = s3.buckets.filter(Prefix="myapp-")

Answer 1

The high level collection s3.buckets.filter(Filters=somefilter) only work for ways that document under describe_tags Filters (list). In such case, you MUST tag your bucket (s3.BucketTagging) before you can use the very specific filtering method s3.buckets.filter(Filters=formatted_tag_filter) (http://boto3.readthedocs.org/en/latest/reference/services/ec2.html#EC2.Client)

IMHO, tagging is a MUST if you plan to manage any resource inside AWS.

Currently, you can do this

s3 = boto3.resource('s3')
for bucket in s3.buckets.all(): 
    if bucket.name.startswith("myapp-"):
        print bucket.name

And following is example code given to filter out KEYS (not bucket) (http://boto3.readthedocs.org/en/latest/guide/collections.html)

# S3 list all keys with the prefix '/photos'
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
    if bucket.name.startswith("myapp-") :
        for obj in bucket.objects.filter(Prefix='/photos'):
            print('{0}:{1}'.format(bucket.name, obj.key))

And there is a warning note using the above example :

Warning Behind the scenes, the above example will call ListBuckets, ListObjects, and HeadObject many times. If you have a large number of S3 objects then this could incur a significant cost.

Answer 2

When you retrieve a list of buckets from the S3 service, you're using the GET / operation on the S3 service.

Docs: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTServiceGET.html

This function does not take any request parameters, so there is no filtering done server-side.

If you want to filter based on your desired prefix, you'll need to retrieve the entire list of buckets, then filter it yourself.