ImportError: No module named _ctypes. Google app engine with bokeh plot

Question

• Python 2.7.13
• Windows 10 64 bit

I've been working through this Udacity web dev course and wanted to try embedding a simple bokeh plot into a web page using this example. Running dev_appserver.py gives the error: ImportError: No module named _ctypes

I have:

• Installed Bokeh and Numpy via pip
• Included Numpy in app.yaml

This answer states Google App Engine doesn't allow importing ctypes. However I'm unsure how to confirm if this is the case with Bokeh. Is this error caused by Bokeh importing ctypes? If so is there a work around?

ERROR    2017-01-21 19:14:53,799 wsgi.py:263]
Traceback (most recent call last):
  File "C:\Users\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform
\google_appengine\google\appengine\runtime\wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "C:\Users\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform
\google_appengine\google\appengine\runtime\wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "C:\Users\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform
\google_appengine\google\appengine\runtime\wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "C:\Users\Google Drive\Udacity web development 2017\udacit
y-cs253\bokeh\1_docs_example\main.py", line 2, in <module>
    from bokeh.plotting import figure
  File "C:\Users\Google Drive\Udacity web development 2017\udacit
y-cs253\bokeh\1_docs_example\lib\bokeh\plotting\__init__.py", line 2, in <module
>
    from ..document import Document; Document
  File "C:\Users\Google Drive\Udacity web development 2017\udacit
y-cs253\bokeh\1_docs_example\lib\bokeh\document.py", line 45, in <module>
    from .core.json_encoder import serialize_json
  File "C:\Users\Google Drive\Udacity web development 2017\udacit
y-cs253\bokeh\1_docs_example\lib\bokeh\core\json_encoder.py", line 43, in <modul
e>
    import numpy as np
  File "C:\Users\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform
\google_appengine\google\appengine\tools\devappserver2\python\sandbox.py", line
706, in load_module
    module = self._find_and_load_module(fullname, fullname, [module_path])
  File "C:\Users\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform
\google_appengine\google\appengine\tools\devappserver2\python\sandbox.py", line
447, in _find_and_load_module
    return imp.load_module(fullname, source_file, path_name, description)
  File "D:\Python27\lib\numpy\__init__.py", line 142, in <module>
    from . import add_newdocs
INFO     2017-01-21 19:14:53,859 module.py:806] default: "GET / HTTP/1.1" 500 -
  File "D:\Python27\lib\numpy\add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "D:\Python27\lib\numpy\lib\__init__.py", line 8, in <module>
    from .type_check import *
  File "D:\Python27\lib\numpy\lib\type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "D:\Python27\lib\numpy\core\__init__.py", line 33, in <module>
    from . import _internal  # for freeze programs
  File "D:\Python27\lib\numpy\core\_internal.py", line 14, in <module>
    import ctypes
  File "D:\Python27\lib\ctypes\__init__.py", line 7, in <module>
    from _ctypes import Union, Structure, Array
  File "C:\Users\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform
\google_appengine\google\appengine\tools\devappserver2\python\sandbox.py", line
964, in load_module
    raise ImportError('No module named %s' % fullname)
ImportError: No module named _ctypes

app.yaml:

runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /.*
  script: main.app

libraries:
- name: jinja2
  version: latest
- name: numpy
  version: latest

main.py

import os, webapp2, jinja2
from bokeh.plotting import figure
from bokeh.embed import components

plot = figure()
plot.circle([1,2], [3,4])
script, div = components(plot)

template_dir = os.path.join(os.path.dirname(__file__), 'templates')
jinja_env = jinja2.Environment(loader = jinja2.FileSystemLoader(template_dir),
                                autoescape = True)

class Handler(webapp2.RequestHandler):
    def write(self, *a, **kw):
        self.response.write(*a, **kw)

    def render_str(self, template, **kw):
        t = jinja_env.get_template(template)
        return t.render(kw)

    def render(self, template, **kw):
        self.write(self.render_str(template, **kw))

class MainPage(Handler):
    def get(self):
        self.render("chart.html", script = script, div = div)

app = webapp2.WSGIApplication([
    ('/', MainPage),
], debug=True)

chart.html

{% extends "base.html" %}

{% block content %}

<!-- Load BokehJS -->
<link
    href="http://cdn.bokeh.org/bokeh/release/bokeh-0.12.0.min.css"
    rel="stylesheet" type="text/css">
<link
    href="http://cdn.bokeh.org/bokeh/release/bokeh-widgets-0.12.0.min.css"
    rel="stylesheet" type="text/css">

<script src="http://cdn.bokeh.org/bokeh/release/bokeh-0.12.0.min.js"></script>
<script src="http://cdn.bokeh.org/bokeh/release/bokeh-widgets-0.12.0.min.js"></script>

{{ script }}
{{ div }}

{% endblock %}

base.html

<!DOCTYPE html>
<html>
<head>
    <title>Udacity Templates!</title>
</head>
<body style="margin: 0">
    <h1 style="background-color: #ddd; color: #888; margin: 0; height: 50px">Udacity Templates</h1>
    {% block content %}
    {% endblock %}
</body>
</html>

gcloud version

• Google Cloud SDK 139.0.1
• app-engine-python 1.9.49
• bq 2.0.24
• bq-win 2.0.24
• bundled-python 2.7.10
• core 2017.01.12
• core-win 2016.11.07
• gcloud
• gsutil 4.22
• gsutil-win 4.20
• powershell 1.0.0.1
• windows-ssh-tools 2016.05.13

UPDATE 1: I have uninstalled numpy 1.12.0 and installed numpy 1.6.1. I now get this error:

NP_MS_DELTA = np.timedelta64(1, 'ms')
TypeError: function takes at most 1 argument (2 given)

This states numpy 1.6.1 can't specify unit in scalar constructor. Does this mean Bokeh relies on numpy>1.6.1?

ERROR    2017-01-22 10:37:45,980 wsgi.py:263]
Traceback (most recent call last):
  File "C:\Users\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform
\google_appengine\google\appengine\runtime\wsgi.py", line 240, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "C:\Users\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform
\google_appengine\google\appengine\runtime\wsgi.py", line 299, in _LoadHandler
    handler, path, err = LoadObject(self._handler)
  File "C:\Users\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform
\google_appengine\google\appengine\runtime\wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
  File "C:\Users\Google Drive\Udacity web development 2017\udacit
y-cs253\bokeh\1_docs_example\main.py", line 2, in <module>
    from bokeh.plotting import figure
  File "C:\Users\Google Drive\Udacity web development 2017\udacit
y-cs253\bokeh\1_docs_example\lib\bokeh\plotting\__init__.py", line 2, in <module
>
    from ..document import Document; Document
  File "C:\Users\Google Drive\Udacity web development 2017\udacit
y-cs253\bokeh\1_docs_example\lib\bokeh\document.py", line 45, in <module>
    from .core.json_encoder import serialize_json
  File "C:\Users\Google Drive\Udacity web development 2017\udacit
y-cs253\bokeh\1_docs_example\lib\bokeh\core\json_encoder.py", line 53, in <modul
e>
    NP_MS_DELTA = np.timedelta64(1, 'ms')
TypeError: function takes at most 1 argument (2 given)
INFO     2017-01-22 10:37:46,170 module.py:806] default: "GET / HTTP/1.1" 500 -

UPDATE 2:

This reminds me of when I pip installed Bokeh. When downloading it's dependency numpy it seemed to rely on numpy>=1.7.1:

Collecting numpy>=1.7.1 (from Bokeh)

Answer 1

I managed to get the dev server running using 2 hacks

ImportError: No module named _ctypes

This is actually caused by flask New Flask uses library named click which uses ctypes Gae doesn't allow ctypes Solution: Install and older version of click with

pip install --target lib --upgrade click==5.1

This fixes ctypes but causes another error

ImportError: No module named msvcrt

This can be easily fixed by adding this lines to appengine_config.py (located in the same folder as app.yaml )

import os, sys

on_appengine = os.environ.get('SERVER_SOFTWARE','').startswith('Development')
if on_appengine and os.name == 'nt':
    sys.platform = "Not Windows"

After this dev servers starts and works

Answer 2

work around issued from google here:

https://issuetracker.google.com/issues/38290292

• goto <sdk_root>\google\appengine\tools\devappserver2\python\sandbox.py

• find the definition of _WHITE_LIST_C_MODULES = [xxx]
  add following two lines to the list:

  '_winreg',
  '_ctypes',
  

• try your app again.

worked for me.

Answer 3

Edit: Please see the answer below for a new workaround from Google.

---

I can state categorically that Bokeh itself does not use ctypes directly, anywhere in the library. But it does use NumPy, and it seems that at least some versions of NumPy do use ctypes? This link:

http://kawahara.ca/using-numpy-on-google-app-engine-with-the-anaconda-python-distribution/

seems to suggest that only version 1.6 of NumPy is supported on GAE. I might assume this is either because that version does not use ctypes, or because Google has specifically whitelisted that version as acceptable somehow.

So the suggestion would be specifically install NumPy 1.6, and not the latest version (either using pip or conda or whatever).