Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

aws cli s3 sync, exclude not working

I am trying to sync data from ec2 to s3 bucket with exclude option

root@ir:ls /data/
f1 f2 f3 
root@ir:aws s3 sync /data/ s3://data/ --profile s3to --exclude "/data/f1/*"
root@ir:aws s3 sync /data/ s3://data/ --profile s3to --exclude "/data/f1/"
root@ir:aws s3 sync /data/ s3://data/ --profile s3to --exclude "/data/f1*"
root@ir:aws s3 sync /data/ s3://data/ --profile s3to --exclude "f1/*"

root@ir:aws --version
aws-cli/1.9.15 Python/2.7.6 Linux/3.13.0-48-generic botocore/1.3.15

But none of above options work and f1 continues to sync to the S3 bucket.

like image 293
Mudasar Yasin Avatar asked Apr 22 '16 10:04

Mudasar Yasin


1 Answers

None of the answers are explicitly stating why the --exclude parameter was seemingly not working.

The Why

--exclude uses a relative path to the directory being sync'd.

For example, the following fails because the sync command is syncing the /data/ directory, and the --exclude parameter is an absolute path.

aws s3 sync /data/ s3://data/ --exclude "/data/f1/*"

The solution is to use a relative path from the folder being sync'd to the folder being excluded with the --exclude parameter:

aws s3 sync /data/ s3://data/ --exclude "f1/*"

Note The Differences

   Fails: --exclude "/data/f1/*"
Succeeds: --exclude "f1/*"
like image 79
Metro Smurf Avatar answered Sep 21 '22 13:09

Metro Smurf