How to store data in GCS while accessing it from GAE and 'GCE' locally

Question

There's a GAE project using the GCS to store/retrieve files. These files also need to be read by code that will run on GCE (needs C++ libraries, so therefore not running on GAE).

In production, deployed on the actual GAE > GCS < GCE, this setup works fine. However, testing and developing locally is a different story that I'm trying to figure out.

As recommended, I'm running GAE's dev_appserver with GoogleAppEngineCloudStorageClient to access the (simulated) GCS. Files are put in the local blobstore. Great for testing GAE.

Since these is no GCE SDK to run a VM locally, whenever I refer to the local 'GCE', it's just my local development machine running linux. On the local GCE side I'm just using the default boto library (https://developers.google.com/storage/docs/gspythonlibrary) with a python 2.x runtime to interface with the C++ code and retrieving files from the GCS. However, in development, these files are inaccessible from boto because they're stored in the dev_appserver's blobstore.

Is there a way to properly connect the local GAE and GCE to a local GCS?

For now, I gave up on the local GCS part and tried using the real GCS. The GCE part with boto is easy. The GCS part is also able to use the real GCS using an access_token so it uses the real GCS instead of the local blobstore by:

cloudstorage.common.set_access_token(access_token) 

According to the docs:

access_token: you can get one by run 'gsutil -d ls' and copy the   str after 'Bearer'. 

That token works for a limited amount of time, so that's not ideal. Is there a way to set a more permanent access_token?

Answer 1

There is convenience option to access Google Cloud Storage from development environment. You should use client library provided with Google Cloud SDK. After executing gcloud init locally you get access to your resources.

As shown in examples to Client library authentication:

# Get the application default credentials. When running locally, these are # available after running `gcloud init`. When running on compute # engine, these are available from the environment. credentials = GoogleCredentials.get_application_default()  # Construct the service object for interacting with the Cloud Storage API - # the 'storage' service, at version 'v1'. # You can browse other available api services and versions here: #     https://developers.google.com/api-client-library/python/apis/ service = discovery.build('storage', 'v1', credentials=credentials)