Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Downloading pattern matched entries from S3 bucket

I have a S3 bucket in which there are several log files stored having the format index.log.yyyy-mm-dd-01 index.log.yyyy-mm-dd-02 . . .

yyyy for year, mm for month and dd for date.

Now i want to download only a few of them. I saw Downloading an entire S3 bucket?. The accepted answer of this post is working absolutely fine if I want to download the entire bucket but what should I do if I want to do some pattern matching? I tried the following commands but they didn't worked:

aws s3 sync s3://mybucket/index.log.2014-08-01-* .
aws s3 sync 's3://mybucket/index.log.2014-08-01-*' .

I also tried using s3cmd for downloading purpose using http://fosshelp.blogspot.in/2013/06 article's POINT 7 and http://s3tools.org/s3cmd-sync. Following were the commands that I ran:

s3cmd -c myconf.txt get --exclude '*.log.*' --include '*.2014-08-01-*' s3://mybucket/ .
s3cmd -c myconf.txt get --exclude '*.log.*' --include '*.2014-08-01-*' s3://mybucket/ .

and a few more permutations of this.

Can anyone tell me why isn't pattern matching happening? Or if there is any other tool that I need to use.

Thanks !!

like image 655
Shrikant Kakani Avatar asked Aug 01 '14 18:08

Shrikant Kakani


People also ask

How do I download data from S3 bucket?

To download an entire bucket to your local file system, use the AWS CLI sync command, passing it the s3 bucket as a source and a directory on your file system as a destination, e.g. aws s3 sync s3://YOUR_BUCKET . . The sync command recursively copies the contents of the source to the destination.

How can you download an S3 bucket including all folders and files?

aws s3 sync s3://mybucket . will download all the objects in mybucket to the current directory. This will download all of your files using a one-way sync. It will not delete any existing files in your current directory unless you specify --delete , and it won't change or delete any files on S3.


1 Answers

Found the solution for the problem. Although I don't know that why other commands were not working.. Solution is as follows:

aws s3 sync s3://mybucket . --exclude "*" --include "*.2014-08-01-*"

Note: --exclude "*" should come before --include "---", doing the reverse won't print anything since it will execute 'exclude' after 'include' (unable to find the reference now where I read this).

like image 149
Shrikant Kakani Avatar answered Oct 15 '22 22:10

Shrikant Kakani