How to replicate Amazon EBS to S3?

Question

We have a site where users upload files, some of them quite large. We've got multiple EC2 instances and would like to load balance them. Currently, we store the files on an EBS volume for fast access. What's the best way to replicate the files so they can be available on more than one instance?

My thought is that some automatic replication process that uploads the files to S3, and then automatically downloads them to other EC2 instances would be ideal.

EBS snapshots won't work because they replicate the entire volume, and we need to be able to replicate the directories of individual customers on demand.

Answer 1

You could write a shell script that would spawn s3cmd to sync your local filesystem with a S3 bucket whenever a new file is uploaded (or deleted). It would look something like:

s3cmd sync  ./  s3://your-bucket/