Menu
I recently used aws s3 sync to mirror a small server (about 4.5M files, 12TB). If I were to run aws s3 sync again, would I get charged for another ~4.5M requests or would only the files that are updated/new be counted as requests?
In general, how are requests counted when using aws s3 sync?
501
The s3 sync command synchronizes the contents of a bucket and a directory, or the contents of two buckets. Typically, s3 sync copies missing or outdated files or objects between the source and target.
It only copies files that have been added or changed since the last sync. It is designed as a one-way sync, not a two-way sync. Your file is being overwritten because the file in the Source is not present in the Destination. This is correct behavior.
Amazon S3 now provides increased performance to support at least 3,500 requests per second to add data and 5,500 requests per second to retrieve data, which can save significant processing time for no additional charge.
S3 cp – Will read all the files from the source location and write into the new location. S3 sync – Will scan the new location and only overwrite the file from the source location if the file is newly created or updated(via file size and modified timestamp comparison).
sync will list all necessary objects, taking as many paginated list requests as necessary. Each list returns 1000 objects. It then does any required gets/puts, but only for objects that have been updated as far as it can tell based on the response elements from the list calls. So in your case there will be ~4,500 list calls + one get/put for each object which has changed.
104
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
