What is the fastest way to duplicate google storage bucket?

Question

I have one 10TB bucket and need to create it's copy as quickly as possible. What is the fastest and most effective way of doing this?

Answer 1

One simple way is to use Google's Cloud Storage Transfer Service. It may also be the fastest, though I have not confirmed this.

Answer 2

Assuming you want to copy the bucket to another bucket in the same location and storage class, you could run gsutil rsync on a GCE instance:

gsutil -m rsync -r -d -p gs://source-bucket gs://dest-bucket

If you want to copy across locations or storage classes the above command will still work, but it will take longer because in that case the data (not just metadata) need to be copied.

Either way, you should check the result status and re-run the rsync command if any errors occurred. (The rsync command will avoid re-copying objects that have already been copied.) You should repeat the rsync command until the bucket has successfully been fully copied.

Answer 3

You can achieve this easily with gsutil.

gsutil -m cp -r gs://source-bucket gs://duplicate-bucket

Are you copying within Google Cloud Storage to a bucket with the same location and storage class? If so, this operation should be very fast. If the buckets have different locations and/or storage classes, the operation will be slower (and more expensive), but this will still be the fastest way.