Python: Upload file to Google Cloud Storage using URL

Question

I have a URL (https://example.com/myfile.txt) of a file and I want to upload it to my bucket (gs://my-sample-bucket) on Google Cloud Storage.

What I am currently doing is:

1. Downloading the file to my system using the requests library.

2. Uploading that file to my bucket using python function.

Is there any way I can upload the file directly using the URL.

Answer 1

You can use urllib2 or requests library to get the file from HTTP, then your existing python code to upload to Cloud Storage. Something like this should work:

import urllib2
from google.cloud import storage

client = storage.Client()

filedata = urllib2.urlopen('http://example.com/myfile.txt')
datatoupload = filedata.read()

bucket = client.get_bucket('bucket-id-here')
blob = Blob("myfile.txt", bucket)
blob.upload_from_string(datatoupload)

It still downloads the file into memory on your system, but I don't think there's a way to tell Cloud Storage to do that for you.