How can I delete old files by name in S3 bucket?

Question

Much like in S3-Bucket/Management/Lifecycles using prefixes, I'd like to prune old files that have certain words.

I'm looking to remove files that start with Screenshot or has screencast in the filename older than 365 days.

Examples

• /Screenshot 2017-03-19 10.11.12.png
• folder1/Screenshot 2019-03-01 14.31.55.png
• folder2/sub_folder/project-screencast.mp4

I'm currently testing if lifecycle prefixes work on files too.

Answer 1

You can write a program to do it, such as this Python script:

import boto3

s3 = boto3.client('s3', region_name='ap-southeast-2')
response = s3.list_objects_v2(Bucket='my-bucket')

keys_to_delete = [{'Key': object['Key']} 
                  for object in response['Contents'] 
                  if object['LastModified'] < datetime(2018, 3, 20)
                     and ('Screenshot' in object['Key'] or 'screencast' in object['Key'])
                 ]

s3.delete_objects(Bucket='my-bucket', Delete={'Objects': keys_to_delete})

You could modify it to be "1 year ago" rather than a specific date.

Answer 2

I don't believe that you can apply lifecycle rules with wildcards such as *screencast*, only with prefixes such as "taxes/" or "taxes/2010".

For your case, I would probably write a script (or perhaps an Athena query) to filter an S3 Inventory report for those files that match your name/age conditions, and then prune them.

Of course, you could write a program to do this as @John Rotenstein suggests. The one time that might not be ideal is if you have millions or billions of objects because the time to enumerate the list of objects would be significant. But it would be fine for a reasonable number of objects.