How to list _all_ objects in Amazon S3 bucket?

Question

S3Client.ListObjects return only 1000 of objects. How to retrieve list of all existing objects using Amazon C# library?

Answer 1

As stated already, Amazon S3 indeed requires Listing Keys Using the AWS SDK for .NET:

As buckets can contain a virtually unlimited number of keys, the complete results of a list query can be extremely large. To manage large result sets, Amazon S3 uses pagination to split them into multiple responses. Each list keys response returns a page of up to 1,000 keys with an indicator indicating if the response is truncated. You send a series of list keys requests until you have received all the keys.

The mentioned indicator is the NextMarker property from the ObjectsResponse Class - its usage is illustrated in the complete example Listing Keys Using the AWS SDK for .NET, with the relevant fragment being:

static AmazonS3 client; client = Amazon.AWSClientFactory.CreateAmazonS3Client(                     accessKeyID, secretAccessKeyID);  ListObjectsRequest request = new ListObjectsRequest(); request.BucketName = bucketName; do {    ListObjectsResponse response = client.ListObjects(request);     // Process response.    // ...     // If response is truncated, set the marker to get the next     // set of keys.    if (response.IsTruncated)    {         request.Marker = response.NextMarker;    }    else    {         request = null;    } } while (request != null); 

Answer 2

Be aware that the answer above is not using the recommended API to List Objects: http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html

The following snippet shows how it looks with the new API:

using (var s3Client = new AmazonS3Client(Amazon.RegionEndpoint.USEast1)) {     ListObjectsV2Request request = new ListObjectsV2Request     {           BucketName = bucketName,           MaxKeys = 10     };     ListObjectsV2Response response;     do     {          response = await s3Client.ListObjectsV2Async(request);           // Process response.          // ...           request.ContinuationToken = response.NextContinuationToken;      } while (response.IsTruncated);         }