Uploading Files to S3 using AWS CLI

Question

I am new to AWS CLI and have requirement to upload files to S3 from a Linux server. Below is my use-case:

Input directory has structure where folder is created on server based on the current date like: /users/data/20200329 and /users/data/20200330.

Each folder contains multiple sub folders:

/users/data/20200329/govt/inbound/truey.gz   
/users/data/20200329/corp/inbound/672695.gz
/users/data/20200330/govt/inbound/dddd.gz   
/users/data/20200330/corp/inbound/4444.gz

I want to upload to S3 like:

s3://bucketname/20200329/truey.gz
s3://bucketname/20200329/672695.gz
s3://bucketname/20200330/dddd.gz
s3://bucketname/20200330/4444.gz

and likewise.

So in nutshell I don't want subfolders in S3 but just single date folder which would contain all files from sub folders mentioned in Input directory. How do I achieve this with the AWS CLI?

Answer 1

This would deal with all the files for a single day:

aws s3 cp /users/data/20200329/govt/inbound/ \
          s3://bucketname/20200329 \
          --recursive 
aws s3 cp /users/data/20200329/corp/inbound/ \
          s3://bucketname/20200329 \
          --recursive 

If you want to iterate over days of the months and so on:

#!/bin/bash

#requires Bash 4.0+ to pad leading 0 in day number
for day in {01..31};
do
   aws s3 cp /users/data/20200329/govt/inbound/ \
            s3://bucketname/202003$day \
            --recursive 
   aws s3 cp /users/data/20200329/corp/inbound/ \
            s3://bucketname/202003$day \
            --recursive 
done

Answer 2

The logic for putting files into specific paths is beyond what the AWS CLI can determine.

You would need to issue individual commands:

aws s3 cp /users/data/20200329/govt/inbound/truey.gz s3://bucketname/20200329/truey.gz 
aws s3 cp /users/data/20200329/corp/inbound/672695.gz s3://bucketname/20200329/672695.gz
aws s3 cp /users/data/20200330/govt/inbound/dddd.gz s3://bucketname/20200330/dddd.gz  
aws s3 cp /users/data/20200330/corp/inbound/4444.gz s3://bucketname/20200330/4444.gz

If you wanted to apply some form of logic for where files should be uploaded, I would recommend writing a script (eg in Python) that will:

• List the input files
• Use some string manipulation to extract the relevant bits
• Issues a upload_file() API call to upload the file to the desired location