How to copy files between S3 buckets in 2 different accounts using boto3

Question

I'm trying to files from a vendors S3 bucket to my S3 bucket using boto3. I'm using the sts service to assume a role to access the vendor s3 bucket. I'm able to connect to the vendor bucket and get a listing of the bucket. I run into CopyObject operation: Access Denied error when copying to my bucket. Here is my script

session = boto3.session.Session(profile_name="s3_transfer")
sts_client = session.client("sts", verify=False)
assumed_role_object = sts_client.assume_role(
    RoleArn="arn:aws:iam::<accountid>:role/assumedrole",
    RoleSessionName="transfer_session",
    ExternalId="<ID>",
    DurationSeconds=18000,
)

creds = assumed_role_object["Credentials"]
src_s3 = boto3.client(
    "s3",
    aws_access_key_id=creds["AccessKeyId"],
    aws_secret_access_key=creds["SecretAccessKey"],
    aws_session_token=creds["SessionToken"],
    verify=False,
)
paginator =src_s3.get_paginator("list_objects_v2")
# testing with just 2 items.
# TODO: Remove MaxItems once script works.
pages = paginator.paginate(
    Bucket="ven_bucket", Prefix="client", PaginationConfig={"MaxItems": 2, "PageSize": 1000}
)
dest_s3 = session.client("s3", verify=False)
for page in pages:
    for obj in page["Contents"]:
        src_key = obj["Key"]
        des_key = dest_prefix + src_key[len(src_prefix) :]
        src = {"Bucket": "ven_bucket", "Key": src_key}
        print(src)
        print(des_key)
        dest_s3.copy(src, "my-bucket", des_key, SourceClient=src_s3)

The line dest_s3.copy... is where I get the error. I have the following policy of my aws user to allow copy to my bucket

{
    "Version": "2012-10-17",
   "Statement": [
    {
        "Sid": "VisualEditor1",
        "Effect": "Allow",
        "Action": [
            "s3:*"
        ],
        "Resource": [
            "arn:aws:s3:::my-bucket/*",
            "arn:aws:s3:::my-bucket/"
        ]
    }
    ]
}

I get the following error when running the above script.

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

Answer 1

The CopyObject() command can be used to copy objects between buckets without having to upload/download. Basically, the two S3 buckets communicate with each other and transfer the data.

This command can also be used to copy between buckets that in different regions and different AWS accounts.

If you wish to copy between buckets that belong to different AWS accounts, then you will need to use a single set of credentials that have:

• GetObject permission on the source bucket
• PutObject permission on the destination bucket

Also, please note that the CopyObject() command is sent to the destination account. The destination bucket effectively pulls the objects from the source bucket.

From your description, your code is assuming a role from the other account to gain read permission on the source bucket. Unfortunately, this is not sufficient for the CopyObject() command because the command must be sent to the destination bucket. (Yes, it is a little hard to discern this from the documentation. That is why the source bucket is specifically named, rather than the destination bucket.)

Therefore, in your situation, to be able to copy the objects, you will need to use a set of credentials from Account-B (the destination) that also has permission to read from Bucket-A (the source). This will require the vendor to modify the Bucket Policy associated with Bucket-A.

If they do not wish to do this, then your only option is to download the objects using the assumed role and then separately upload the files to your own bucket using credentials from your own Account-B.