How to download a file from the internet to a Google Cloud bucket directly

Question

I want to download a file over 20GB from the internet into a google cloud bucket directly. Just like doing in a local command line the following:

wget http://some.url.com/some/file.tar 

I refuse to download the file to my own computer and then copying the file to the bucket using:

gsutil cp file.tar gs://the-bucket/

For the moment I am trying (just at this very moment) to use datalab to download the file and then copying the file from there to the bucket.

Answer 1

A capability of the Google Cloud Platform as it relates to Google Cloud Storage is the functional area known as "Storage Transfer Service". The documentation for this is available here.

At the highest level, this capability allows you to define a source of data that is external to Google such as data available as a URL or on AWS S3 storage and then schedule that to be copied to Google Cloud Storage in the background. This function seems to perform the task you want ... the data is copied from an Internet source to GCS directly.

---

A completely different story would be the realization that GCP itself provides compute capabilities. What this means is that you can run your own logic on GCP through simple mechanisms such as a VM, Cloud Functions or Cloud Run. This helps us in this story by realizing that we could execute our code to download the Internet based data from within GCP itself to a local temp file. This file could then be uploaded into GCS from within GCP. At no time did the data that will end up in GCP ever go anywhere than from the source to Google. Once retrieved from the source, the transfer rate of the data from the GCP compute to GCS storage should be optimal as it is passing exclusively over Googles internal ultra high speed networks.