Best way to move files between S3 buckets?

Question

I'd like to copy some files from a production bucket to a development bucket daily.

For example: Copy productionbucket/feed/feedname/date to developmentbucket/feed/feedname/date

Because the files I want are so deep in the folder structure, it's too time consuming to go to each folder and copy/paste.

I've played around with mounting drives to each bucket and writing a windows batch script, but that is very slow and it unnecessarily downloads all the files/folders to the local server and back up again.

Answer 1

Update

As pointed out by alberge (+1), nowadays the excellent AWS Command Line Interface provides the most versatile approach for interacting with (almost) all things AWS - it meanwhile covers most services' APIs and also features higher level S3 commands for dealing with your use case specifically, see the AWS CLI reference for S3:

• sync - Syncs directories and S3 prefixes. Your use case is covered by Example 2 (more fine grained usage with --exclude, --include and prefix handling etc. is also available):

  The following sync command syncs objects under a specified prefix and bucket to objects under another specified prefix and bucket by copying s3 objects. [...]

  aws s3 sync s3://from_my_bucket s3://to_my_other_bucket 

For completeness, I'll mention that the lower level S3 commands are also still available via the s3api sub command, which would allow to directly translate any SDK based solution to the AWS CLI before adopting its higher level functionality eventually.

---

Initial Answer

Moving files between S3 buckets can be achieved by means of the PUT Object - Copy API (followed by DELETE Object):

This implementation of the PUT operation creates a copy of an object that is already stored in Amazon S3. A PUT copy operation is the same as performing a GET and then a PUT. Adding the request header, x-amz-copy-source, makes the PUT operation copy the source object into the destination bucket. Source

There are respective samples for all existing AWS SDKs available, see Copying Objects in a Single Operation. Naturally, a scripting based solution would be the obvious first choice here, so Copy an Object Using the AWS SDK for Ruby might be a good starting point; if you prefer Python instead, the same can be achieved via boto as well of course, see method copy_key() within boto's S3 API documentation.

PUT Object only copies files, so you'll need to explicitly delete a file via DELETE Object still after a successful copy operation, but that will be just another few lines once the overall script handling the bucket and file names is in place (there are respective examples as well, see e.g. Deleting One Object Per Request).