Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

awscli s3 sync wildcards

I'm trying to sync all files in a directory that start with "model.ckpt" to an S3 bucket path, by trying this:

aws s3 sync ./model.ckpt* $S3_CKPT_PATH

But I'm getting the error:

Unknown options: ./model.ckpt-0.meta,<my S3_CKPT_PATH path>

However, aws s3 sync . $S3_CKPT_PATH works, but gives me a lot of additional files I don't want.

Anybody know how I can do this?

like image 994
Austin Avatar asked Apr 10 '19 00:04

Austin


People also ask

Can you use wildcard in aws CLI?

Currently AWS CLI doesn't provide support for UNIX wildcards in a command's “path” argument. However, it is quite easy to replicate this functionality using the --exclude and --include parameters available on several aws s3 commands.

Does aws S3 sync overwrite?

It only copies files that have been added or changed since the last sync. It is designed as a one-way sync, not a two-way sync. Your file is being overwritten because the file in the Source is not present in the Destination. This is correct behavior.

Is aws S3 sync recursive?

Syncs directories and S3 prefixes. Recursively copies new and updated files from the source directory to the destination. Only creates folders in the destination if they contain one or more files.


1 Answers

When using aws s3 sync, all files in a folder are included.

If you wish to specify wildcards, you will need to Use Exclude and Include Filters.

For example:

aws s3 sync mydir s3://bucket/folder/ --exclude "*" --include "model.ckpt*"
like image 181
John Rotenstein Avatar answered Sep 18 '22 14:09

John Rotenstein