How to get more than 1000 objects from S3 by using list_objects_v2?

Question

I have more than 500,000 objects on s3. I am trying to get the size of each object. I am using the following python code for that

import boto3  bucket = 'bucket' prefix = 'prefix'  contents = boto3.client('s3').list_objects_v2(Bucket=bucket,  MaxKeys=1000, Prefix=prefix)["Contents"]  for c in contents:     print(c["Size"]) 

But it just gave me the size of the top 1000 objects. Based on the documentation we can't get more than 1000. Is there any way I can get more than that?

Answer 1

The inbuilt boto3 Paginator class is the easiest way to overcome the 1000 record limitation of list-objects-v2. This can be implemented as follows

s3 = boto3.client('s3')  paginator = s3.get_paginator('list_objects_v2') pages = paginator.paginate(Bucket='bucket', Prefix='prefix')  for page in pages:     for obj in page['Contents']:         print(obj['Size']) 

For more details: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Paginator.ListObjectsV2