How to delete files recursively from an S3 bucket

Question

With the latest aws-cli python command line tools, to recursively delete all the files under a folder in a bucket is just:

aws s3 rm --recursive s3://your_bucket_name/foo/

Or delete everything under the bucket:

aws s3 rm --recursive s3://your_bucket_name

If what you want is to actually delete the bucket, there is one-step shortcut:

aws s3 rb --force s3://your_bucket_name

which will remove the contents in that bucket recursively then delete the bucket.

Note: the s3:// protocol prefix is required for these commands to work

This used to require a dedicated API call per key (file), but has been greatly simplified due to the introduction of Amazon S3 - Multi-Object Delete in December 2011:

Amazon S3's new Multi-Object Delete gives you the ability to delete up to 1000 objects from an S3 bucket with a single request.

See my answer to the related question delete from S3 using api php using wildcard for more on this and respective examples in PHP (the AWS SDK for PHP supports this since version 1.4.8).

Most AWS client libraries have meanwhile introduced dedicated support for this functionality one way or another, e.g.:

Python

You can achieve this with the excellent boto Python interface to AWS roughly as follows (untested, from the top of my head):

import boto
s3 = boto.connect_s3()
bucket = s3.get_bucket("bucketname")
bucketListResultSet = bucket.list(prefix="foo/bar")
result = bucket.delete_keys([key.name for key in bucketListResultSet])

Ruby

This is available since version 1.24 of the AWS SDK for Ruby and the release notes provide an example as well:

bucket = AWS::S3.new.buckets['mybucket']

# delete a list of objects by keys, objects are deleted in batches of 1k per
# request.  Accepts strings, AWS::S3::S3Object, AWS::S3::ObectVersion and 
# hashes with :key and :version_id
bucket.objects.delete('key1', 'key2', 'key3', ...)

# delete all of the objects in a bucket (optionally with a common prefix as shown)
bucket.objects.with_prefix('2009/').delete_all

# conditional delete, loads and deletes objects in batches of 1k, only
# deleting those that return true from the block
bucket.objects.delete_if{|object| object.key =~ /\.pdf$/ }

# empty the bucket and then delete the bucket, objects are deleted in batches of 1k
bucket.delete!

Or:

AWS::S3::Bucket.delete('your_bucket', :force => true)

You might also consider using Amazon S3 Lifecycle to create an expiration for files with the prefix foo/bar1.

Open the S3 browser console and click a bucket. Then click Properties and then LifeCycle.

Create an expiration rule for all files with the prefix foo/bar1 and set the date to 1 day since file was created.

Save and all matching files will be gone within 24 hours.

Just don't forget to remove the rule after you're done!

No API calls, no third party libraries, apps or scripts.

I just deleted several million files this way.

A screenshot showing the Lifecycle Rule window (note in this shot the Prefix has been left blank, affecting all keys in the bucket):

With s3cmd package installed on a Linux machine, you can do this

s3cmd rm s3://foo/bar --recursive

In case if you want to remove all objects with "foo/" prefix using Java AWS SDK 2.0

import java.util.ArrayList;
import java.util.Iterator;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.*;

//...

ListObjectsRequest listObjectsRequest = ListObjectsRequest.builder()
    .bucket(bucketName)
    .prefix("foo/")
    .build()
;
ListObjectsResponse objectsResponse = s3Client.listObjects(listObjectsRequest);

while (true) {
    ArrayList<ObjectIdentifier> objects = new ArrayList<>();

    for (Iterator<?> iterator = objectsResponse.contents().iterator(); iterator.hasNext(); ) {
        S3Object s3Object = (S3Object)iterator.next();
        objects.add(
            ObjectIdentifier.builder()
                .key(s3Object.key())
                .build()
        );
    }

    s3Client.deleteObjects(
        DeleteObjectsRequest.builder()
            .bucket(bucketName)
            .delete(
                Delete.builder()
                    .objects(objects)
                    .build()
            )
            .build()
    );

    if (objectsResponse.isTruncated()) {
        objectsResponse = s3Client.listObjects(listObjectsRequest);
        continue;
    }

    break;
};

The voted up answer is missing a step.

Per aws s3 help:

Currently, there is no support for the use of UNIX style wildcards in a command's path arguments. However, most commands have --exclude "<value>" and --include "<value>" parameters that can achieve the desired result......... When there are multiple filters, the rule is the filters that appear later in the command take precedence over filters that appear earlier in the command. For example, if the filter parameters passed to the command were --exclude "*" --include "*.txt" All files will be excluded from the command except for files ending with .txt

aws s3 rm --recursive s3://bucket/ --exclude="*" --include="/folder_path/*"