Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

`aws s3 cp` vs `aws s3 sync` behavior and cost [closed]

I have a static site that I am deploying to s3 and then using CloudFront to distribute to users. After I build the site, I want to push the new build to s3. I found 2 approaches to do that.

  • aws s3 cp --recursive ./public/ s3://bucket-name --cache-control 'public, max-age=300, s-maxage=31536000'

  • aws s3 sync --delete ./public/ s3://bucket-name --cache-control 'public, max-age=300, s-maxage=31536000'

I am planning to deploy once or twice every week.

I want to know which of these is less expensive (money)? To be more clear, I want to know which among these will cost me less in the long run?

I tried reading the docs, but I was not able to figure out the differences. Please help me with this.

like image 557
Rohan Singh Avatar asked Nov 07 '20 13:11

Rohan Singh


People also ask

What is the difference between S3 CP and S3 sync commands?

S3 cp – Will read all the files from the source location and write into the new location. S3 sync – Will scan the new location and only overwrite the file from the source location if the file is newly created or updated(via file size and modified timestamp comparison).

What is aws S3 sync command?

The s3 sync command synchronizes the contents of a bucket and a directory, or the contents of two buckets. Typically, s3 sync copies missing or outdated files or objects between the source and target.

How does aws S3 CP work?

The cp command initiates a copy operation to or from Amazon S3. The --recursive option instructs the AWS CLI for Amazon S3 to descend into subdirectories on the source. The --quiet option instructs the AWS CLI for Amazon S3 to print only errors rather than a line for each file copied.

How do you make S3 sync faster?

To reduce latency, reduce the geographical distance between the instance and your Amazon S3 bucket. If the instance is in the same Region as the source bucket, then set up an Amazon Virtual Private Cloud (Amazon VPC) endpoint for S3. VPC endpoints can help improve overall performance.


1 Answers

One thing to note is that aws s3 cp --recursive and aws s3 sync --delete have different behaviors.

aws s3 cp will copy all files, even if they already exist in the destination area. It also will not delete files from your destination if they are deleted from the source.

aws s3 sync looks at the destination before copying files over and only copies over files that are new and updated. The --delete flag also will delete things at the destination if they were removed in source.

The sync command is what you want as it is designed to handle keeping two folders in sync while copying the minimum amount of data. Sync should result in less data being pushed into S3 bucket so that should have a less cost overall.

To give a counterexample, a use case where aws s3 cp outperforms and is lower cost than sync is if you just need to transfer files and you know all the files are new to the destination. This is more performant and lower cost because the code is not checking the destination if things exist before starting the transfer.

like image 152
JD D Avatar answered Oct 20 '22 21:10

JD D