How can you update the storage class for all items in an S3 bucket?

Question

I've found myself wishing I could easily change all items in a bucket to a particular storage class on S3. Often this is because items were uploaded in Standard, and I want them in Reduced Redundancy to save a few bucks.

I don't see a way to do this through the AWS Console.

What is the best way to update all the files in a bucket?

Answer 1

As you are talking about the entire bucket, I think the best way to do this would be with creating a life cycle rule. That can be done through the console:

1. Open the AWS console, go to the relevant bucket and click on "Management".
2. Click on "Lifecycle" and then on "Add Lifecycle rule".
3. Give it a name and click on next.

4. Select whether you want it to run on current versions or on previous versions (If versioning is configured for this bucket), and then select "Single Zone IA".

5. Click on next three times, and you're done.

Or you could do it through the AWS-CLI like so:

Create a json file called lifecycle.json like so:

{
          "Rules": [
              {
                  "ID": "Move all objects to one zone infrequent access",
                  "Prefix": "",
                  "Status": "Enabled",
                  "Transitions": [
                      {
                          "Days": 30,
                          "StorageClass": "ONEZONE_IA"
                      }
                  ]
              }
        ]
}

and then run:

aws s3api put-bucket-lifecycle-configuration --bucket <Bucket name> --lifecycle-configuration file://./lifcycle.json

In transition I placed 30 days as it is, currently, the minimum time required for an object to exist before being transitioned to one zone IA.

Answer 2

Use the awscli pip package

https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html

Example:

aws s3 cp \
  --storage-class STANDARD_IA \
  --region='us-west-2' \
  --recursive \
  s3://myBucket/logs/ s3://myBucket/logs/

Answer 3

There is no way to do this via the AWS Console. You will need to iterate over them and update the meta data on each object.

Here's a ruby script that does just that:

https://gist.github.com/mcfadden/b1e564f3323f98720ff2

A few other thoughts:

Set the correct storage class on object creation. You won't want to loop through all the items again.

Some Storage Classes aren't available for all objects. For example, you can't set objects to the Standard - Infrequent Access class until they've been in the bucket for 30 days.

If you are trying to use the Standard - Infrequent Access storage class, you can set up a lifecycle rule to automatically move objects to this storage class after 30 days.