Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Copying directly from S3 to Google Cloud Storage

I can migrate data from Amazon AWS S3 to Azure using AWS SDK for Java and Azure SDk for Java. Now I want to do migrate data from Amazon AWS S3 to Google Cloud storage using Java.

like image 353
leaveme_alone Avatar asked Jan 29 '14 17:01

leaveme_alone


People also ask

How do I transfer data from S3 to GCP?

Click on “Create transfer” While selecting the source system click on “Amazon S3 bucket” As for Amazon S3 bucket name use storage-transfer-service-test-gcp or whichever name you have used before. For Access Key ID and Secret Access Key insert the ones that we have generated before in the AWS console.

Is Google cloud storage compatible with S3?

XML API. The Cloud Storage XML API is interoperable with some tools and libraries that work with services such as Amazon Simple Storage Service (Amazon S3).

How do I transfer data from S3 bucket to local?

You can use cp to copy the files from an s3 bucket to your local system. Use the following command: $ aws s3 cp s3://bucket/folder/file.txt .


1 Answers

The gsutil command-line tool supports S3. After you've configured gsutil, you'll see this in your ~/.boto file:

# To add aws credentials ("s3://" URIs), edit and uncomment the
# following two lines:
#aws_access_key_id = 
#aws_secret_access_key = 

Fill in the aws_access_key_id and aws_secret_access_key settings with your S3 credentials and uncomment the variables.

Once that's set up, copying from S3 to GCS is as easy as:

gsutil cp -R s3://bucketname gs://bucketname

If you have a lot of objects, run with the -m flag to perform the copy in parallel with multiple threads:

gsutil -m cp -R s3://bucketname gs://bucketname
like image 106
jterrace Avatar answered Oct 16 '22 02:10

jterrace