Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can we copy the files and folders recursively between aws s3 buckets using boto3 Python?

Is it possible to copy all the files in one source bucket to other target bucket using boto3. And source bucket doesn't have regular folder structure.

Source bucket: SRC
Source Path: A/B/C/D/E/F..
where in D folder it has some files,
E folder has some files

Target bucket: TGT
Target path: L/M/N/

I need to copy all the files and folders from above SRC bucket from folder C to TGT bucket under N folder using boto3.

Can any one aware of any API or do we need to write new python script to complete this task.

like image 448
Gowthaman Javi Avatar asked Apr 17 '17 19:04

Gowthaman Javi


People also ask

How do I copy files from one S3 bucket to another S3 bucket in Python?

It allows users to create, and manage AWS services such as EC2 and S3. You can use the Boto3 Session and bucket. copy() method to copy files between S3 buckets. You need your AWS account credentials for performing copy or move operations.

How do I transfer files between S3 buckets in two different accounts with boto3?

The CopyObject() command can be used to copy objects between buckets without having to upload/download. Basically, the two S3 buckets communicate with each other and transfer the data. This command can also be used to copy between buckets that in different regions and different AWS accounts.

What is the command to copy files recursively in a folder to an S3 bucket?

When passed with the parameter --recursive the aws s3 cp command recursively copies all objects from source to destination. It can be used to download and upload large set of files from and to S3.

Can you copy data from one S3 bucket to another?

Depending on your use case, you can perform the data transfer between buckets using one of the following options: Run parallel uploads using the AWS Command Line Interface (AWS CLI) Use an AWS SDK. Use cross-Region replication or same-Region replication.


1 Answers

S3 store object, it doesn't store folder, even '/' or '\' is part of the object key name. You just need to manipulate the key name and copy the data over.

import boto3
old_bucket_name = 'SRC'
old_prefix = 'A/B/C/'
new_bucket_name = 'TGT'
new_prefix = 'L/M/N/'
s3 = boto3.resource('s3')
old_bucket = s3.Bucket(old_bucket_name)
new_bucket = s3.Bucket(new_bucket_name)

for obj in old_bucket.objects.filter(Prefix=old_prefix):
    old_source = { 'Bucket': old_bucket_name,
                   'Key': obj.key}
    # replace the prefix
    new_key = obj.key.replace(old_prefix, new_prefix, 1)
    new_obj = new_bucket.Object(new_key)
    new_obj.copy(old_source)

Optimized technique of defining new_key suggested by zvikico:

new_key = new_prefix + obj.key[len(old_prefix):]
like image 149
mootmoot Avatar answered Sep 19 '22 13:09

mootmoot