Google Cloud Storage on Appengine Dev Server

Question

There's a similar question that was recently responded to on Stackoverflow here: Google Cloud Storage Client not working on dev appserver

The solution was to either upgrade the SDK to 1.8.8 or use the previous revision of the GCS client library which didn't have the bug instead.

I'm currently using 1.8.8 and have tried downloading multiple revisions and /_ah/gcs doesn't load for me. After using up a significant number of my backend instances trying to understand how GCS and app engine work together, it'd be great if I could just test it on my local server instead!

When I visit localhost:port/_ah/gcs I get a 404 not found error.

Just a heads up, to install the library all I did was drag and drop the code into my app folder. I'm wondering if maybe I skipped a setup step? I wasn't able to find the answer in the documentation!

thanks!!

Note To clarify this is my first week using GCS, so my first time trying to use the dev_server to host it.

Answer 1

I was able to find the google cloud storage files I wrote to a bucket locally at:

    localhost:port/_ah/gcs/bucket_name/file_suffix

Where port is by default 8080, and the file was written to: /bucket_name/file_suffix

For those trying to understand the full process of setting up a simple python GAE app and testing local writes to google cloud storage:

1. Follow the google app engine "quickstart":

https://cloud.google.com/appengine/docs/standard/python/quickstart

2. Run a local dev server with:

    dev_appserver.py app.yaml 

3. If using python, follow "App Engine and Google Cloud Storage Sample":

https://cloud.google.com/appengine/docs/standard/python/googlecloudstorageclient/app-engine-cloud-storage-sample

If you run into "ImportError: No module named cloudstorage" you need to create a file named appengine_config.py

    touch appengine_config.py

and add to it:

    from google.appengine.ext import vendor
    vendor.add('lib')

GAE runs this script automatically when starting your local dev server with dev_appserver.py app.yaml, and it is necessary to run this script for GAE to find the cloudstorage library in your lib/ folder

4. "Writing a file to cloud storage" from the same tutorial:

    def create_file(self, filename):
    """Create a file."""

       self.response.write('Creating file {}\n'.format(filename))

       # The retry_params specified in the open call will override the default
       # retry params for this particular file handle.
       write_retry_params = cloudstorage.RetryParams(backoff_factor=1.1)
       with cloudstorage.open(
           filename, 'w', content_type='text/plain', options={
               'x-goog-meta-foo': 'foo', 'x-goog-meta-bar': 'bar'},
               retry_params=write_retry_params) as cloudstorage_file:
                   cloudstorage_file.write('abcde\n')
                   cloudstorage_file.write('f'*1024*4 + '\n')
       self.tmp_filenames_to_clean_up.append(filename) 

       with cloudstorage.open(
           filename, 'w', content_type='text/plain', options={
               'x-goog-meta-foo': 'foo', 'x-goog-meta-bar': 'bar'},
               retry_params=write_retry_params) as cloudstorage_file:
                   cloudstorage_file.write('abcde\n')
                   cloudstorage_file.write('f'*1024*4 + '\n')

Where filename is /bucket_name/file_suffix

4. After calling create_file via a route in your WSGI app, your file will be available at:

    localhost:port/_ah/gcs/bucket_name/file_suffix

Where port is by default 8080, and the file was written to: /bucket_name/file_suffix

Postscript

Unfortunately, I did not find either 3) or 4) in their docs, so I hope this helps someone get set up more easily in the future.

Answer 2

To access gcs objects on dev_appserver, you must specify the bucket & object name, i.e. /_ah/gcs/[bucket]/[object].