Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Move/copy data from one folder to another on AWS S3

I am looking for all the methods for moving/copying the data from one folder to another on AWS S3 bucket.


Method 1: Via AWS CLI (Most easy)

Download and install awscli on ur instance, I am using here windows(64-bit link) and run "asw configure" to fill up your configuration and just run this single command on cmd

aws s3 cp s3://from-source/ s3://to-destination/ --recursive

Here cp for copy and recursive to copy all files


Method 2: Via AWS CLI using python

import os
import awscli
if os.environ.get('LC_CTYPE', '') == 'UTF-8':
    os.environ['LC_CTYPE'] = 'en_US.UTF-8'

from awscli.clidriver import create_clidriver
driver = create_clidriver()
driver.main('s3 mv s3://staging/AwsTesting/research/    s3://staging/AwsTesting/research_archive/ --recursive'.split())

Even this worked for me perfectly


Method 3: Via Boto using python

import boto3
s3 = boto3.resource('s3')
copy_source = {
    'Bucket': 'staging',
    'Key': '/AwsTesting/research/'
}
s3.meta.client.copy(copy_source, 'staging', '/AwsTesting/research_archive/')

With my understanding I have assumed the 'key' for bucket is just the folder prefix so I have mentioned the folder path here

Error:

Invalid bucket name "s3://staging": Bucket name must match the regex "^[a-zA-Z0-9.-_]{1,255}$"

Even I changed it to simple bucket name as "staging" but no success.


How can I understand bucket connectivity via boto and the concept of this key?

like image 349
Shivkumar kondi Avatar asked Jun 06 '17 07:06

Shivkumar kondi


People also ask

Can you copy data from one S3 bucket to another?

Depending on your use case, you can perform the data transfer between buckets using one of the following options: Run parallel uploads using the AWS Command Line Interface (AWS CLI) Use an AWS SDK. Use cross-Region replication or same-Region replication.

What is the command to copy files recursively in a folder to an S3 bucket?

When passed with the parameter --recursive the aws s3 cp command recursively copies all objects from source to destination. It can be used to download and upload large set of files from and to S3.

How do I move objects in S3 bucket?

To move objectsNavigate to the Amazon S3 bucket or folder that contains the objects that you want to move. Select the check box to the left of the names of the objects that you want to move. Choose Actions and choose Move from the list of options that appears.


2 Answers

import boto3
s3 = boto3.resource('s3')
copy_source = {
    'Bucket': 'staging',
    'Key': 'AwsTesting/research/filename.csv'
}
s3.meta.client.copy(copy_source, 'staging', 'AwsTesting/')
like image 105
raisashi Avatar answered Oct 24 '22 10:10

raisashi


An alternative to using cp with the CLI is sync - https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

aws s3 sync s3://mybucket s3://mybucket2

It will essentially do the same thing.

like image 35
Saxon Avatar answered Oct 24 '22 10:10

Saxon