syncing AWS S3 bucket to EC2 Servers

Question

I am trying to use rsync to sync my S3 bucket with my EC2 servers. However I am having trouble coming up with the code. On my EC2 server I have tried the following, but it doesn't work. I know my S3 address is wrong but I'm not sure what to put in its place. iosSourceCode is the bucket name. How can I sync the files in this bucket to my EC2 server's files? After I get this to work I was going to set up a cronjob to do this every 10 minutes or whatever. Is there a better way to do this and if so how? Please provide code, thanks!

sudo rsync -ra iosSourceCode.s3-website-us-east-1.amazonaws.com /var/www/

Answer 1

You can use this command:

aws s3 --region <your region name> sync s3://<your bucket name> /your/directory/path

So in your case:

aws s3 --region us-east-1 sync s3://osSourceCode.s3-website /var/www/

This is a one-way sync (only downloads updates from S3 in this example) so if you want to sync both ways then you need two commands with the source and destination swapped around:

aws s3 --region us-east-1 sync s3://osSourceCode.s3-website /var/www/
aws s3 --region us-east-1 sync /var/www s3://osSourceCode.s3-website

You can add this to a crontab entry to make it occur periodically, as per the following example:

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/opt/aws/bin
MAILTO=root
HOME=/

*/2 * * * * root aws s3 sync --region ap-southeast-2 /var/www/html s3://mywebsitecode/ >> /usr/local/logs/aws.log 2>&1
*/5 * * * * root aws s3 sync --region ap-southeast-2 s3://mywebsitecode/ /var/www/html >> /usr/local/logs/aws.log 2>&1

Answer 2

Please use s3cmd

http://s3tools.org/s3cmd

use s3cmd sync

syntax will be as below

s3cmd sync s3://mybucket/myfolder/files/ /var/mybucket/myfolder/files/

You can put above syntax in shell script and add script to cron to run it as specific time interval.