How to get size of filtered objectsCollection in boto3

Question

I've tried the following to get the len/content_length of the s3.Bucket.objectsCollection in boto3 v1.7.37:

import boto3    
s3 = boto3.resource('s3')
bucket = s3.Bucket('myBucket')
bucketObjects = bucket.objects.filter(Prefix='myPrefix')
if (len(bucketObjects) > 0):
     do_something()
else:
     do_something_else()

Unfortunately this gives me the following error:

TypeError: object of type 's3.Bucket.objectsCollection' has no len()

I've also tried this with bucketobjects.content_length and got

AttributeError: 's3.Bucket.objectsCollection' object has no attribute 'content_length'

Am I going to have to iterate through the list and count the objects or is there a better way?

Answer 1

As Leo K said bucket.objects.filter returns iterable object that have no definite length. But you could limit the iteration by using the limit method. So in case you wish to check there is at list one item found you could use the following:

results = bucket.objects.filter(Prefix=prefix_filter) if list(results.limit(1)): do_something() else: do_something_else()

Answer 2

bucket.objects.filter() (and most other high-level boto3 calls that return collections of objects) return iterable objects that have no definite length. This is deliberate, because the potential size of the lists can be very large. To prevent your app running out of memory, they are never loaded all at once - so the size remains unknown until you actually ask for each item in turn and get to the very end.

If you know for sure the list is short, get it all with bucketObjects = list(bucketObjects), but it is better not to do this. Count them by reading one at a time:

c=0
for b in bucketObjects: c+=1

(if you really need maximum efficiency, try the low-level api: client=boto3.client('s3') and get large chunks of the list in a few calls or if you're sure it isn't too big get the whole list at once; but this is rarely worth it)