How to select a file from aws s3 by using wild character

Question

I have many a files in s3 bucket and I want to copy those files which have start date of 2012. This below command copies all the file.

aws s3 cp s3://bp-dev/bp_source_input/ C:\Business_Panorama\nts\data\in --recursive  --include "201502_nts_*.xlsx"

Answer 1

After doing many rounds of check and getting help from bsnchan, I am able to use exclude and include command in aws s3 cli. Please make sure that you put the spaces correctly.

for copy specific file:

aws s3 cp s3://itx-agj-cons-ww-bp-dev/bp_source_input/ C:\Business_Panorama\nts\data\in  --recursive --exclude "*" --include "*%mth_cd%_%source%_all.xlsx"

(Note mth_cd is parameter used in bat file)

For checking of file existance.

aws s3 ls s3://itx-agj-cons-ww-bp-dev/bp_source_input/ --recursive | FINDSTR  "201502_nts_.*.xlsx"

(Note: windows cli, for unix it will be grep)

Thanks a lot.

Answer 2

You may want to add the "--exclude" flag before your include filter.

The AWS CLI takes the filter "--include" to include it in your already existing search. Since all the files are being returned, you need to exclude all the files first, before including the 2015*.xlsx.

If you want files with only the format "201502_nts_*.xlsx", you can run aws s3 cp s3://bp-dev/bp_source_input/ C:\Business_Panorama\nts\data\in --recursive --exclude * --include "201502_nts_*.xlsx"