I am looking for all the methods for moving/copying the data from one folder to another on AWS S3 bucket.
Method 1: Via AWS CLI (Most easy)
Download and install awscli on ur instance, I am using here windows(64-bit link) and run "asw configure" to fill up your configuration and just run this single command on cmd
aws s3 cp s3://from-source/ s3://to-destination/ --recursive
Here cp for copy and recursive to copy all files
Method 2: Via AWS CLI using python
import os
import awscli
if os.environ.get('LC_CTYPE', '') == 'UTF-8':
os.environ['LC_CTYPE'] = 'en_US.UTF-8'
from awscli.clidriver import create_clidriver
driver = create_clidriver()
driver.main('s3 mv s3://staging/AwsTesting/research/ s3://staging/AwsTesting/research_archive/ --recursive'.split())
Even this worked for me perfectly
Method 3: Via Boto using python
import boto3
s3 = boto3.resource('s3')
copy_source = {
'Bucket': 'staging',
'Key': '/AwsTesting/research/'
}
s3.meta.client.copy(copy_source, 'staging', '/AwsTesting/research_archive/')
With my understanding I have assumed the 'key' for bucket is just the folder prefix so I have mentioned the folder path here
Error:
Invalid bucket name "s3://staging": Bucket name must match the regex "^[a-zA-Z0-9.-_]{1,255}$"
Even I changed it to simple bucket name as "staging" but no success.
How can I understand bucket connectivity via boto and the concept of this key?
Depending on your use case, you can perform the data transfer between buckets using one of the following options: Run parallel uploads using the AWS Command Line Interface (AWS CLI) Use an AWS SDK. Use cross-Region replication or same-Region replication.
When passed with the parameter --recursive the aws s3 cp command recursively copies all objects from source to destination. It can be used to download and upload large set of files from and to S3.
To move objectsNavigate to the Amazon S3 bucket or folder that contains the objects that you want to move. Select the check box to the left of the names of the objects that you want to move. Choose Actions and choose Move from the list of options that appears.
import boto3
s3 = boto3.resource('s3')
copy_source = {
'Bucket': 'staging',
'Key': 'AwsTesting/research/filename.csv'
}
s3.meta.client.copy(copy_source, 'staging', 'AwsTesting/')
An alternative to using cp
with the CLI is sync
- https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
aws s3 sync s3://mybucket s3://mybucket2
It will essentially do the same thing.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With