aws cli copy between S3 regions on EC2

Question

I am trying to copy between two S3 buckets in different regions using the Command Line Interface on an EC2 server.

region info:
EC2 instance: us-west-2
S3 origin: us-east-1
S3 destination: us-west-2

The following commands work perfectly from the EC2 server:
aws s3 cp s3://n-virginia/origin s3://n-virginia/destination --recursive --source-region us-east-1 --region us-east-1 --profile my_profile

aws s3 cp s3://oregon/origin s3://oregon/destination --recursive --source-region us-west-2 --region us-west-2 --profile my_profile

I need to run the following command from the EC2 server:
aws s3 cp s3://n-virginia/origin s3://oregon/destination --recursive --source-region us-east-1 --region us-west-2 --profile my_profile

If I run that command from a local machine it works, but if I run it from the EC2 server that I used for the previous two commands I get the following error:

Error:"A client error (AccessDenied) occurred when calling the CopyObject operation: VPC endpoints do not support cross-region requests"

I am able to copy the files from the origin bucket to the EC2 server, and then copy from the EC2 server to the destination bucket, but this is not an acceptable solution in production. I don't understand why it will work on a local machine but not on the EC2 server ("my_profile" is identical on both machines)

Answer 1

As pointed out in the comments the problem is your VPC has an endpoint and cross region copies are not supported.

To fix that, either temporarily disable the VPC endpoint, by updating your VPC route table, or just create a new VPC without a VPC endpoint and launch an EC2 there.

Cross region replication would be ideal, but as pointed out, that only effects new items in the bucket

Instead of using aws s3 cp you probably want to use aws s3 sync. Sync will only copy changed files, thus allowing you to rerun it again in case it is interrupted. For example:

aws s3 sync s3://n-virginia/origin s3://oregon/destination

Note also that both cp and sync do NOT preserve ACL. So if you have changed ACL permission on individual files they will all be set to the default after the copy. There are some other tools that are supposed to preserve ACL the like https://s3tools.org which seems to work for me.

Answer 2

I know this is an old post but we have faced the same issues recently.

To update the @astrotom response, Amazon S3 Cross-Region Replication (CRR) now supports copying existing objects. you just need to ask for aws support team to unlock the feature. full explanation here and here

From our side, we preferred @brendan solution even though it saturates the network. you can find here a Kubernetes job that can help you automate it.

you can find in this blog multiple approaches to migrate our buckets cross region cross account