Moving many files in the same bucket

Question

I've got 200k files in a bucket which I need to move into a sub folder within the same bucket, whats the best approach?

Answer 1

I recently encountered the same problem. I solved it using the command line API.

http://docs.aws.amazon.com/cli/latest/index.html http://docs.aws.amazon.com/cli/latest/reference/s3/mv.html

aws s3 mv s3://BUCKETNAME/myfolder/photos/ s3://BUCKETNAME/myotherfolder/photos/ --recursive 

I had a need for the objects to be publicly viewable, so I added the --acl public-read option.

Answer 2

Recently was able to do this with one command. Went much faster than individual requests for each file too.

Running a snippet like this:

aws s3 mv s3://bucket-name/ s3://bucket-name/subfolder --recursive --exclude "*" --include "*.txt" 

Use the --include flag to selectively pick up the files you want

Answer 3

There is no 'Rename' operation though it would be great if there was.

Instead, you need to loop through each item that you want to rename, perform a copy to a new object and then a delete on the old object.

• http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectCOPY.html
• http://docs.amazonwebservices.com/AmazonS3/latest/API/RESTObjectDELETE.html

Note: for simplistic purposes I'm assuming you don't have versioning enabled on your bucket.