copy file from Google Drive to Google Cloud Storage within Google

Question

Is it possible to copy a file from Google Drive to Google Cloud Storage? I imagine it would be very fast since both are on a similar storage system.

I haven't seen any information on either ways to do this seamlessly, without having to download the file (probably to an out-of-google system) and then re-uploading it. About one year ago, a similar question was asked: Transfer files from dropbox/drive to Google cloud storage.

Is there a way to do this directly, with the file staying within-Google the entire time?

Answer 1

@frunkad commented a link to a nice workaround, but for completeness I will recite this here, as it is currently the top result in search.

You can open a Colab (Juypiter Notebook on Google servers), mount your gDrive and use the gCloud CLI to copy files.

1. open https://colab.research.google.com/ and connect to a machine
2. Mount your Drive by executing the code below, clicking on the link and pasting the authentification key

from google.colab import drive
drive.mount(‘/content/drive’)

3. Connect to your Google Cloud Storage. (Project id can be found here)

from google.colab import auth
auth.authenticate_user()
project_id = 'your-project-id'
!gcloud config set project {project_id}
!gsutil ls

4. Copy files using gsutil. Use -m tag for multi-threading to increase speed. (There is a Subfolder called "My Drive" that you have to address in your mounted drive)

bucket_name = 'your_bucket_name'
!gsutil -m cp -r /content/drive/My\ Drive/Your-Data/* gs://{bucket_name}/

Original Author Philip Lies

Link to his colab: https://colab.research.google.com/drive/1Xc8E8mKC4MBvQ6Sw6akd_X5Z1cmHSNca

Answer 2

There's no direct way to do this. You could write an App Engine or Compute Engine app that reads from Drive using its API and writes to GCS using its API.