Upload CSV file to Google Cloud Storage using Python

Question

I need to automate uploading local csv file into Google Cloud storage bucket in Python. Which Python library can I use? Any sample code would be much appreciated.

Answer 1

We can use the google python client api to upload files to google cloud storage.

First, install the api client as follows.

>pip install --upgrade google-api-python-client

Then, enable api authentication to get application default credentials.

>gcloud beta auth application-default login

Below is a sample code which uploads a local file to google cloud storage using application default credentials.

from googleapiclient import discovery
from oauth2client.client import GoogleCredentials

credentials = GoogleCredentials.get_application_default()
service = discovery.build('storage', 'v1', credentials=credentials)

filename = 'C:\\MyFiles\\sample.csv'
bucket = 'my_bucket'

body = {'name': 'dest_file_name.csv'}
req = service.objects().insert(bucket=bucket, body=body, media_body=filename)
resp = req.execute()

This will upload the file inside my_bucket. The full google storage url for the uploaded file would be gs://my_bucket/dest_file_name.csv

Answer 2

Another way is as shown in this link.

First, connect to cloud

from gcloud import storage
client = storage.Client()

Then select the bucket and choose remote filename

bucket = client.get_bucket('<your-bucket-name>')
blob = bucket.blob('remote_file.txt')

Finally, upload the local file. I prefer the following way but there are alternative ways.

blob.upload_from_filename('local_file_txt')

If you have a variable, the above line requires you to write your variable into disk, then upload which may not be the best way. Instead you can directly write to the blob from a string.

blob.upload_from_string('this is test content!')