Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to download an entire bucket in GCP?

I have a problem downloading entire folder in GCP. How should I download the whole bucket? I run this code in GCP Shell Environment:

gsutil -m cp -R gs://my-uniquename-bucket ./C:\Users\Myname\Desktop\Bucket

and I get an error message: "CommandException: Destination URL must name a directory, bucket, or bucket subdirectory for the multiple source form of the cp command. CommandException: 7 files/objects could not be transferred."

Could someone please point out the mistake in the code line?

like image 997
Justas Avatar asked Oct 27 '19 17:10

Justas


People also ask

Which command is used to download multiple files GCP?

I would suggest downloading the files with gsutil . However if you have a large number of files to transfer you might want to use the gsutil -m option, to perform a parallel (multi-threaded/multi-processing) copy: gsutil -m cp -R gs://your-bucket .

How do I download dataset from Google Cloud?

In the Explorer panel, expand your project and dataset, then select the table. In the details panel, click Export and select Export to Cloud Storage. In the Export table to Google Cloud Storage dialog: For Select Google Cloud Storage location, browse for the bucket, folder, or file where you want to export the data.

How do I download a folder from the Google Cloud?

Upload and downloading files and folders Files and folders can only be uploaded to and downloaded from your home directory. In your Cloud Shell Editor Explorer, right-click a directory or file and then click Copy Download Link, Download, or Upload Files. Alternatively, you can navigate to File > Download/Upload Files.

How do I download from GCP VM?

In the Google Cloud console, go to the VM instances page. In the list of virtual machine instances, click SSH in the row of the instance that you want to connect to. After the connection is established, click the download icon download. The download dialog opens.


2 Answers

To download an entire bucket You must install google cloud SDK

then run this command

gsutil -m cp -R gs://project-bucket-name path/to/local

where path/to/local is your path of local storage of your machine

like image 88
Mussa Charles Avatar answered Sep 18 '22 18:09

Mussa Charles


The error lies within the destination URL as specified by the error message.

I run this code in GCP Shell Environment

Remember that you are running the command from the Cloud Shell and not in a local terminal or Windows Command Line. Thus, it is throwing that error because it cannot find the path you specified. If you inspect the Cloud Shell's file system/structure, it resembles more that of a Unix environment in which you can specify the destination like such instead: ~/bucketfiles/. Even a simple gsutil -m cp -R gs://bucket-name.appspot.com ./ will work since Cloud Shell can identify the ./ directory which is the current directory.

A workaround to this issue is to perform the command on your Windows Command Line. You would have to install Google Cloud SDK beforehand.

Alternatively, this can also be done in Cloud Shell, albeit with an extra step:

  1. Download the bucket objects by running gsutil -m cp -R gs://bucket-name ~/ which will download it into the home directory in Cloud Shell
  2. Transfer the files downloaded in the ~/ (home) directory from Cloud Shell to the local machine either through the User Interface or by running gcloud alpha cloud-shell scp
like image 37
JKleinne Avatar answered Sep 19 '22 18:09

JKleinne