S3 Sync vs. Cross-region Replication

Question

I have ~1.5PB data in S3 us-west-1. I want to copy this to us-east-2 region. Should I use cross-region replication or S3 Sync? And, what are the pros and cons of using the two options?

I researched a few AWS threads and found that they describe each one in great detail (E.g. https://aws.amazon.com/premiumsupport/knowledge-center/s3-large-transfer-between-buckets/ and https://aws.amazon.com/premiumsupport/knowledge-center/s3-bucket-migrate-region/), without explaining the difference between the two.

Please note that our security policies don't allow Snowball Edge.

Can someone please help me?

Answer 1

When you adds the replication to your the bucket then the Objects that existed before will not be copied to the other bucket. Replication will also not let you replicate if Objects created with server-side encryption using customer-provided (SSE-C) encryption keys. for more detail you should read this.

So in this case, either you can use the AWS S3 Sync or AWCCLi's cp command (will be slower) or use Snowball Edge (Which you can do't do as per the description)

aws s3 cp --recursive s3://<bucket>>
aws s3 sync s3://<bucket> s3://<bucket>>

AWS Sync is good for small size objects/buckets but as you mentioned you have peta bytes of data then I will provide you two solutions:

1. S3 batch Operations: You can use Amazon S3 batch operations to copy multiple objects with a single request.
2. S3DistCp: The S3DistCp operation on Amazon EMR can perform parallel copying of large volumes of objects across Amazon S3 buckets. More Read

Once you have copied your data to another S3 bucket you can enable the replication which will replicate all new objects.

Notes: These solutions can be expensive, so make sure you read about the cost if using these operations.

Answer 2

Replication will copy newly PUT objects into the destination bucket.

Sync will copy existing objects to the destination bucket.

Generally you would enable replication and then run sync once to copy the existing objects.