Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Downloading the latest file in an S3 bucket using AWS CLI? [duplicate]

I have an S3 bucket that contains database backups. I am creating a script to download the latest backup (and eventually restore it somewhere else), but I'm not sure how to go about only grabbing the most recent file from a bucket.

Is it possible to copy only the most recent file from an S3 bucket to a local directory using AWS CLI tools?

like image 564
Abe Miessler Avatar asked Jul 14 '16 21:07

Abe Miessler


People also ask

How do I download from S3 bucket to aws CLI?

You can use cp to copy the files from an s3 bucket to your local system. Use the following command: $ aws s3 cp s3://bucket/folder/file.txt .

Can you duplicate an S3 bucket?

Create bucket, enter the bucket name, choose region, copy settings from existing bucket. Create bucket. Once bucket created, go to the source bucket to which you want to copy the files from. Select all (if needed or else you can choose desired files and folders), Actions > Copy.

How do I download data from S3 bucket?

To download an entire bucket to your local file system, use the AWS CLI sync command, passing it the s3 bucket as a source and a directory on your file system as a destination, e.g. aws s3 sync s3://YOUR_BUCKET . . The sync command recursively copies the contents of the source to the destination.


2 Answers

And here is a bash script create based on @error2007s's answer. This script requires your aws profile and bucket name as variables, and downloads the latest object to your ~/Downloads folder:

#!/bin/sh PROFILE=your_profile BUCKET=your_bucket  OBJECT="$(aws s3 ls --profile $PROFILE $BUCKET --recursive | sort | tail -n 1 | awk '{print $4}')" aws s3 cp s3://$BUCKET/$OBJECT ~/Downloads/$OBJECT --profile $PROFILE 
like image 176
JBaczuk Avatar answered Sep 19 '22 12:09

JBaczuk


The above solutions are using Bash. If one wants to do the same thing in PowerShell for downloading on Windows here is the script:

# This assumes AWS CLI exe is in your path. $s3location = "s3://bucket-name" $files = $(aws s3 ls $s3location --recursive | sort | select -last 3) $dlPath = "C:\TEMP"  foreach ($s3FileInfo in $files) {     $filename = $s3FileInfo.Split()[-1]     $path = "${s3location}/${filename}"     aws s3 cp $path $dlPath     echo("Done downloading ${path} to ${dlPath}") } 
like image 24
Nikki Ganju Avatar answered Sep 19 '22 12:09

Nikki Ganju