aws s3 replace file atomically

Question

Environment

• I copied a file, ./barname.bin, to s3, using the command aws s3 cp ./barname.bin s3://fooname/barname.bin

• I have a different file, ./barname.1.bin that I want to upload in place of that file

---

How can I upload and replace (overwrite) the file at s3://fooname/barname.bin with ./barname.1.bin?

Goals:

• Don't change the s3 url used to access the file (new file should also be available at s3://fooname/barname.bin).
• zero/minimum 'downtime'/unavailability of the s3 link.

Answer 1

As I understand it, you've got an existing file located at s3://fooname/barname.bin and you want to replace it with a new file. To replace that, you should just upload a new one on top of the old one: aws s3 cp ./barname.1.bin s3://fooname/barname.bin.

The old file will be replaced. According to the S3 docs, this is atomic, though due to EC2s replication pattern, requests for the key may still return the old file for some time.

Note (thanks @Chris Kuehl): though the replacement is technically atomic, it's possible for multipart downloads to end up with chunks from different versions of the file. 😬