Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Delete all versions of an object in S3 using python?

I have a versioned bucket and would like to delete the object (and all of its versions) from the bucket. However, when I try to delete the object from the console, S3 simply adds a delete marker but does not perform a hard delete.

Is it possible to delete all versions of the object (hard delete) with a particular key?:

s3resource = boto3.resource('s3')
bucket = s3resource.Bucket('my_bucket')
obj = bucket.Object('my_object_key')

# I would like to delete all versions for the object like so:
obj.delete_all_versions()

# or delete all versions for all objects like so:
bucket.objects.delete_all_versions()
like image 491
rooscous Avatar asked Oct 18 '17 21:10

rooscous


People also ask

How do I clean up my S3 bucket?

To empty an S3 bucketIn the Bucket name list, select the option next to the name of the bucket that you want to empty, and then choose Empty. On the Empty bucket page, confirm that you want to empty the bucket by entering the bucket name into the text field, and then choose Empty.

How to delete all versions of files in S3 versioned bucket?

Delete all versions of all files in s3 versioned bucket using AWS CLI and jq. · GitHub Instantly share code, notes, and snippets. Delete all versions of all files in s3 versioned bucket using AWS CLI and jq. cmd= "aws s3api delete-object --bucket $bucket --key $key --version-id $versionId"

How do I delete an object version in Amazon S3?

Amazon S3 shows all the versions for the object. Select the check box next to the Version ID for the versions that you want to retrieve. Choose Delete . In Permanently delete objects?, enter permanently delete . When you permanently delete an object version, the action cannot be undone.

How does versioning work in Amazon S3?

When versioning is enabled, a simple DELETE cannot permanently delete an object. Instead, Amazon S3 inserts a delete marker in the bucket, and that marker becomes the current version of the object with a new ID.

How to delete an object from S3 using Boto 3 in Python?

In this article, we will see how to delete an object from S3 using Boto 3 library of Python. Example − Delete test.zip from Bucket_1/testfolder of S3 Step 1 − Import boto3 and botocore exceptions to handle exceptions. Step 2 − s3_files_path is parameter in function.


1 Answers

The other answers delete objects individually. It is more efficient to use the delete_objects boto3 call and batch process your delete. See the code below for a function which collects all objects and deletes in batches of 1000:

bucket = 'bucket-name'
s3_client = boto3.client('s3')
object_response_paginator = s3_client.get_paginator('list_object_versions')

delete_marker_list = []
version_list = []

for object_response_itr in object_response_paginator.paginate(Bucket=bucket):
    if 'DeleteMarkers' in object_response_itr:
        for delete_marker in object_response_itr['DeleteMarkers']:
            delete_marker_list.append({'Key': delete_marker['Key'], 'VersionId': delete_marker['VersionId']})

    if 'Versions' in object_response_itr:
        for version in object_response_itr['Versions']:
            version_list.append({'Key': version['Key'], 'VersionId': version['VersionId']})

for i in range(0, len(delete_marker_list), 1000):
    response = s3_client.delete_objects(
        Bucket=bucket,
        Delete={
            'Objects': delete_marker_list[i:i+1000],
            'Quiet': True
        }
    )
    print(response)

for i in range(0, len(version_list), 1000):
    response = s3_client.delete_objects(
        Bucket=bucket,
        Delete={
            'Objects': version_list[i:i+1000],
            'Quiet': True
        }
    )
    print(response)
like image 134
AndrewC Avatar answered Sep 20 '22 05:09

AndrewC