How to import BigQuery in AppEngine for Python

Question

I am trying to run a BigQuery query from Google AppEngine (deployed) using Python 2.7, but I am seeing this error in StackDriver's Error Reporting:

ImportError: No module named cloud

This is my code (main.py):

from __future__ import absolute_import

import webapp2
from google.cloud import bigquery


class MainPage(webapp2.RequestHandler):
    def get(self):

        # Instantiates a client
        bigquery_client = bigquery.Client()

        # The name for the new dataset
        dataset_name = 'my_new_set'

        # Prepares the new dataset
        dataset = bigquery_client.dataset(dataset_name)

        # Creates the new dataset
        dataset.create()

        # Remove unwanted chars
        #self.response.write(str(container))


app = webapp2.WSGIApplication([
    ('/', MainPage),
], debug=True)

This is my (app.yaml):

runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /.*
  script: main.app

The error message would make me assume that the BigQuery's library is not being imported. However, if this code is being deployed in AppEngine, shouldn't the library already be installed in AppEngine by default?

---

Trying to solve the problem

Attempt # 1

I found this post that refers to a similar issue. The suggestion was to add this line to the top of the file. I added the line to my file, but the problem still exists:

from __future__ import absolute_import

Source: No module named cloud while using google.cloud import bigquery

Attempt # 2

I installed BigQuery's client locally in my laptop:

pip install google-cloud-bigquery==0.22.1

I also installed the same client in the "lib" folder to have it uploaded to AppEngine once it is deployed:

pip install --target='lib' google-cloud-bigquery==0.22.1

This last, also requires a file named "appengine_config.py" to be created with this content:

# appengine_config.py
from google.appengine.ext import vendor

    # Add any libraries install in the "lib" folder.
    vendor.add('lib')

Source: https://cloud.google.com/appengine/docs/standard/python/tools/using-libraries-python-27

However, this attempt did not work either. The error message changed to the following:

*File "/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/httplib2/__init__.py", line 352: print('%s:' % h, end=' ', file=self._fp) ^ SyntaxError: invalid syntax
at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google_auth_httplib2.py:23)
at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google/cloud/_helpers.py:31)
at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google/cloud/bigquery/_helpers.py:21)
at <module> (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/lib/google/cloud/bigquery/__init__.py:26)
at get (/base/data/home/apps/p~experimenting-1130/2.400173726395247238/main.py:75)
at dispatch (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:545)
at dispatch (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:547)
at __call__ (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1077)
at default_dispatcher (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1253)
at __call__ (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1505)
at __call__ (/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.3/webapp2.py:1511)*

How can I import the BigQuery library correctly in AppEngine (deployed)?

Thanks for your help.

Answer 1

The following solution worked for me without having to use from __future__ import absolute_import. There are three main steps to follow.

1. Copy google-api-python-client and google-cloud into project folder

Even though it sounds counterintuitive, according to the documentation

[...] Python client libraries are not installed in the App Engine Python runtime environment, [so] they must be vendored into your application just like third-party libraries.

So in order to use google.cloud one must copy the library code into the project's source directory. The library code, along with the application code, is uploaded to App Engine.

To copy a library code into your project:

1. Create a directory (e.g. lib/) to store third-party libraries in your project's root folder

   mkdir lib
   

2. Copy the google-api-python-client and google-cloud libraries into the folder you just created. I use pip in the following example.

   pip install -t lib/ --upgrade google-api-python-client
   pip install -t lib/ --upgrade google-cloud
   

2. Link installed libraries to app

1. Create a file named appengine_config.pyin the same folder as your app.yaml file

2. Edit appengine_config.py and include the following code

   # appengine_config.py
   from google.appengine.ext import vendor
   
   # Add any libraries install in the "lib" folder.
   vendor.add('lib') 
   

3. Include added libraries to requirements.txt

1. Edit your requirements.txt file and include the names of your added libraries

   # other requirements
   google-api-python-client
   google-cloud
   

You should now be able to use from google.cloud import bigquery with no problem after deploying your app.

For more information see using third-party libraries