pyramid + jinja2 and new GAE runtime

Question

I am trying to run Pyramid with Jinja2 using new Python 2.7 runtime in threadsafe mode and GAE 1.6.0 pre-release SDK. I've made modifications to my app as outlined here, i.e. I've set runtime: python27, threadsafe: true in app.yaml and got rid of main() function. When I generate response by myself it works fine, but when I try to bring jinja2 into the equation, I get the following exception:

ERROR    2011-11-07 00:10:34,356 wsgi.py:170] 
Traceback (most recent call last):
  File "/gae/google/appengine/runtime/wsgi.py", line 168, in Handle
  [...]
  File "/myapp/source/myapp-tip/main.py", line 29, in <module>
    config.include('pyramid_jinja2')
  File "/myapp/source/myapp-tip/lib/dist/pyramid/config/__init__.py", line 616, in include
    c(configurator)
  File "lib/dist/pyramid_jinja2/__init__.py", line 390, in includeme
    _get_or_build_default_environment(config.registry)
  File "/lib/dist/pyramid_jinja2/__init__.py", line 217, in _get_or_build_default_environment
    _setup_environment(registry)
  File "/lib/dist/pyramid_jinja2/__init__.py", line 253, in _setup_environment
    package = _caller_package(('pyramid_jinja2', 'jinja2', 'pyramid.config'))
  File "/lib/dist/pyramid_jinja2/__init__.py", line 136, in caller_package
    for t in self.inspect.stack():
  File "/usr/lib/python2.7/inspect.py", line 1056, in stack
    return getouterframes(sys._getframe(1), context)
  File "/usr/lib/python2.7/inspect.py", line 1034, in getouterframes
    framelist.append((frame,) + getframeinfo(frame, context))
  File "/usr/lib/python2.7/inspect.py", line 1009, in getframeinfo
    lines, lnum = findsource(frame)
  File "/usr/lib/python2.7/inspect.py", line 534, in findsource
    module = getmodule(object, file)
  File "/usr/lib/python2.7/inspect.py", line 506, in getmodule
    main = sys.modules['__main__']
KeyError: '__main__'

I tried to mess around a bit with pyramid_jinja2 code to work around this issue, only to be left with another exception:

ERROR    2011-11-04 12:06:38,720 wsgi.py:170] 
Traceback (most recent call last):
  File "/gae/google/appengine/runtime/wsgi.py", line 168, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  [...]
  File "/myapp/source/myapp-tip/main.py", line 29, in <module>
    config.add_jinja2_search_path("templates")
  File "/myapp/source/myapp-tip/lib/dist/pyramid/config/util.py", line 28, in wrapper
    result = wrapped(self, *arg, **kw)
  File "/lib/dist/pyramid_jinja2/__init__.py", line 311, in add_jinja2_search_path
    env.loader.searchpath.append(abspath_from_resource_spec(d))
  File "/myapp/source/myapp-tip/lib/dist/pyramid/asset.py", line 38, in abspath_from_asset_spec
    return pkg_resources.resource_filename(pname, filename)
  File "/myapp/source/myapp-tip/pkg_resources.py", line 840, in resource_filename
    return get_provider(package_or_requirement).get_resource_filename(
  File "/myapp/source/myapp-tip/pkg_resources.py", line 160, in get_provider
    __import__(moduleOrReq)
  File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 640, in Decorate
    return func(self, *args, **kwargs)
  File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 1756, in load_module
    return self.FindAndLoadModule(submodule, fullname, search_path)
  File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 640, in Decorate
    return func(self, *args, **kwargs)
  File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 1628, in FindAndLoadModule
    description)
  File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 640, in Decorate
    return func(self, *args, **kwargs)
  File "/gae/google/appengine/tools/dev_appserver_import_hook.py", line 1571, in LoadModuleRestricted
    description)
ImportError: Cannot re-init internal module __main__

I'd be happy if anybody could shed some light on what pyramid is trying to do under the hood. Judging by the latter stack trace it seems it's trying to resolve an asset, but why is it trying to reload __main__? I'm not even sure my problem is caused by pyramid or GAE.

Thanks for any insight on this issue.

Answer 1

I'm not familiar with pyramid, but the problem really does seem to be with this line:

config.include('pyramid_jinja2')

Whatever that config thing is, it seems to be doing some dynamic import magic.

Don't do that.

The app engine environment doesn't handle imports the way you would in normal python. Step through that line with a debugger and you'll wind up in the replacement version of the import system, which you'll soon see, only implements a small part of what real python does.

If possible, just use a normal import statement... Otherwise, you're going to have to dig into config.include and get it to play nice with the restricted importing features on GAE.

Answer 2

I managed to make it work using Pyramid 1.3's AssetResolver. First attempt is here. Just not sure what the lifetime/scope of the resolver should be in this case, I will figure it out later.