How to restore folders (or entire buckets) to Amazon S3 from Glacier?

Question

If you use s3cmd you can use it to restore recursively pretty easily:

s3cmd restore --recursive s3://mybucketname/ 

I've also used it to restore just folders as well:

s3cmd restore --recursive s3://mybucketname/folder/

If you're using the AWS CLI tool (it's nice, you should), you can do it like this:

aws s3 ls s3://<BUCKET_NAME> --recursive | awk '{print $4}' | xargs -L 1 aws s3api restore-object --restore-request '{"Days":<DAYS>,"GlacierJobParameters":{"Tier":"<TIER>"}}' --bucket <BUCKET_NAME> --key

Replace <BUCKET_NAME> with the bucket name you want and provide restore parameters <DAYS> and <TIER>.

<DAYS> is the number of days you want to restore the object for and <TIER> controls the speed of the restore process and has three levels: Bulk, Standard, or Expedited:

The above answers didn't work well for me because my bucket was mixed with objects on Glacier and some that were not. The easiest thing for me was to create a list of all GLACIER objects in the bucket, then attempt to restore each one individually, ignoring any errors (like already in progress, not an object, etc).

1. Get a listing of all GLACIER files (keys) in the bucket

   aws s3api list-objects-v2 --bucket <bucketName> --query "Contents[?StorageClass=='GLACIER']" --output text | awk '{print $2}' > glacier-restore.txt

2. Create a shell script and run it, replacing your "bucketName".

   #!/bin/sh
   
   for x in `cat glacier-restore.txt`
     do
       echo "Begin restoring $x"
       aws s3api restore-object --restore-request Days=7 --bucket <bucketName> --key "$x"
       echo "Done restoring $x"
     done
   

Credit goes to Josh at http://capnjosh.com/blog/a-client-error-invalidobjectstate-occurred-when-calling-the-copyobject-operation-operation-is-not-valid-for-the-source-objects-storage-class/, a resource I found after trying some of the above solutions.

There isn't a built-in tool for this. "Folders" in S3 are an illusion for human convenience, based on forward-slashes in the object key (path/filename) and every object that migrates to glacier has to be restored individually, although...

Of course you could write a script to iterate through the hierarchy and send those restore requests using the SDKs or the REST API in your programming language of choice.

Be sure you understand how restoring from glacier into S3 works, before you proceed. It is always only a temporary restoration, and you choose the number of days that each object will persist in S3 before reverting back to being only stored in glacier.

Also, you want to be certain that you understand the penalty charges for restoring too much glacier data in a short period of time, or you could be in for some unexpected expense. Depending on the urgency, you may want to spread the restore operation out over days or weeks.