How do I get the size of a boto3 Collection?

Question

The way I have been using is to transform the Collection into a List and query the length:

s3 = boto3.resource('s3') bucket = s3.Bucket('my_bucket') size = len(list(bucket.objects.all())) 

However, this forces resolution of the whole collection and obviates the benefits of using a Collection in the first place. Is there a better way to do this?

Answer 1

There is no way to get the count of keys in a bucket without listing all the objects this is a limitation of AWS S3 (see https://forums.aws.amazon.com/thread.jspa?messageID=164220).

Getting the Object Summaries (HEAD) doesn't get the actual data so should be a relatively inexpensive operation and if you are just discarding the list then you could do:

size = sum(1 for _ in bucket.objects.all()) 

Which will give you the number of objects without constructing a list.