Google cloud storage - Download file from web

Question

I want to use Google cloud storage in my next project. My aim is tracking various web sites and collecting some photos. As, I read the documentation for gsutil; I'm able download the file manually to my server and upload it google cloud storage by using gsutil.

Downloading and uploading files generates so much traffic in my server. Are there a way to let google cloud download file direct from http?

Answer 1

Main answer

This is very easy to do from the Google Cloud Shell. Seems to work for all file sizes:

 curl http://speedtest.tele2.net/10GB.zip | gsutil cp - gs://YOUR_BUCKET_NAME/10GB.zip

Basically curl streams the data directly to the bucket.

Alternative (file smaller than ~ 4.6 GB)

Original approach: This works as long as your download is less than ~ 4.6 GB. Launch the Cloud Shell (first icon on your top right after you login to your project in GCP) and use wget to download the file you want. For instance, to download 7-Zip type:

wget https://www.7-zip.org/a/7z1805-x64.exe

Now with the file in your Cloud Shell user home you can copy it to a Google Cloud Storage bucket using the gsutil command:

gsutil cp ./7z1805-x64.exe gs://your_bucket_name/

Alternative (bigger than ~ 4.6 GB)

If the file is bigger than 4.6 GB you can still do it but you need to mount the bucket in your Cloud Shell using gcsfuse:

Create a directory in your Cloud Shell user home

 mkdir ~/mybucket

Now mount your bucket in that directory using gcsfuse:

 gcsfuse bucket_name ~/mybucket

Change the current directory to mount point directory:

 cd mybucket

(if you want to have some fun run "df -h ." to see how much space you got in that mount point)

Now use wget to get the file directly into your bucket (sample using 10GB file off the web):

 wget https://speed.hetzner.de/10GB.bin

Answer 2

Google Cloud Storage only accepts data directly. There's no way to pass it a URL and have it save the contents as an object.

However, there's no reason you couldn't build this functionality yourself. For example, you could set up one or more dedicated GCE instanceS that would load URLs and then save them to GCS. Google doesn't charge for network ingress into GCE or for from GCE into GCS within a region, either, which helps.