How to use virtualenv with Google App Engine SDK on Mac OS X 10.6

Question

I am pulling my hair out trying to figure this out because I had it working until last week and somehow it broke.

When I setup a virtualenv for a Google App Engine app and start the app with dev_appserver.py, I get errors importing the standard library (like "ImportError: No module named base64").

Here's what I'm doing:

(Using the system Python)

virtualenv --python=python2.5 --no-site-packages ~/.virtualenv/foobar

Then I add the a gae.pth file to ~/.virtualenv/foobar/lib/python2.5/site-packages/ containing the Google App Engine libraries:

/usr/local/google_appengine /usr/local/google_appengine/lib/antlr3 /usr/local/google_appengine/lib/cacerts /usr/local/google_appengine/lib/django /usr/local/google_appengine/lib/fancy_urllib /usr/local/google_appengine/lib/ipaddr /usr/local/google_appengine/lib/webob_1_1_1 /usr/local/google_appengine/lib/yaml/lib 

(That's based on this answer.)

Then I source my "foobar" virtualenv and try to start my app with dev_appserver.py.

The server starts but the first request errors out with the aforementioned "ImportError: No module named base64". If I visit the admin console I get "ImportError: No module named cgi".

If I start up python, I can load these modules.

>>> import base64 >>> base64.__file__ '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/base64.py' 

It seems that the SDK's sandboxing is preventing these libraries from getting loaded. But like I said, I had this working until last week...something changed or I inadvertently broke my virtualenv and I can't figure out how I got it working in the first place.

Software versions:

Google App Engine SDK 1.3.7
Mac OS X Snow Leopard 10.6.4
virtualenv 1.5.1

Update: In response to Alan Franzoni's questions:

I am using the system Python that came with Mac OS X. I installed virtualenv via easy_install. I upgraded to virtualenv 1.5.1 today to try to fix the problem.

If I run python /usr/local/bin/dev_appserver.py with the virtualenv python, the problem persists. If I deactivate the virtualenv and run that command with the system python2.5, it works. (Also, I can use the GoogleAppEngineLauncher to start my app.)

Here is a full stack trace (this one uses the Kay framework, but the problem is the same with webapp):

Traceback (most recent call last):   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 3206, in _HandleRequest     self._Dispatch(dispatcher, self.rfile, outfile, env_dict)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 3149, in _Dispatch     base_env_dict=env_dict)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 525, in Dispatch     base_env_dict=base_env_dict)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 2402, in Dispatch     self._module_dict)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 2312, in ExecuteCGI     reset_modules = exec_script(handler_path, cgi_path, hook)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 2208, in ExecuteOrImportScript     exec module_code in script_module.__dict__   File "/Users/look/myapp/kay/main.py", line 17, in <module>     kay.setup()   File "/Users/look/myapp/kay/__init__.py", line 122, in setup     from google.appengine.ext import db   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1287, in Decorate     return func(self, *args, **kwargs)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1937, in load_module     return self.FindAndLoadModule(submodule, fullname, search_path)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1287, in Decorate     return func(self, *args, **kwargs)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1839, in FindAndLoadModule     description)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1287, in Decorate     return func(self, *args, **kwargs)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver.py", line 1790, in LoadModuleRestricted     description)   File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/db/__init__.py", line 81, in <module>     import base64 ImportError: No module named base64 

Answer 1

It's an issue 4339 with the GAE SDK, it's confirmed and there are two slightly different patches available in the bug entry that make it work.

What happens is dev_appserver.py sets up a restricted python environment by disallowing access to any non-system-python modules and it does that by calculating the system python folder from the location of the os module. In a virtualenv instance the os.py gets symlinked into the virtualenv but gets compiled straight into virtualenv, and this is the path that dev_appserver uses, effectively blocking access to any module from the system python library that is not linked by virtualend, which is most of them. The solution is to "bless" both paths.