Modules paths in Python

Question

I have created a folder with all my modules for my GAE application and with external libraries like Jinja2 to keep everything sorted in one place. I have folders structure like this:

lib\
   \utils\
         \__init__.py
   \firepython
   \jinja2
   \jsonpickle
   __init__.py
   sessions.py

When I try to load Jinja from utils__init__.py, I get error ImportError: No module named jinja2.environment. When I look at Jinja2 imports instructions, I see them look like from jinja2.loaders. I try to change them to be like from lib.jinja2.loaders but some other errors then appear about imports. More than that I don't think it's a good practice to change these imports in external libraries source if there is a more convenient and right way to import modules properly. I also have added some paths to PYTHONPATH but it doesn't solve all problems. How can I properly import an external package that is placed in another folder, may be with a deep structure?

Answer 1

Indeed you should not have to change imports in external libraries - though depending on your environment, you might even have too.

PYTHONPATH

Modifying your PYTHONPATH should suffice; PYTHONPATH should contain a 'lib' path that is either absolute or relative to your home, eg.

Then you could simply do

from jinja2 import WHATEVER

sys.path.append

Another way to go without PYTHONPATH is to use sys.path.append() and add your paths from your python code. I actually favor that, as it also allows to have per-application paths.

use virtualenv

Details would be a bit long to be put here, but please follow the official doc

These options applies to general python development rather than GAE specificities; if it does not work on your development machine you should post more details (exact imports, absolute paths, pythonpath...).

A proper project structure and use of appcfg.py should workout dependencies when uploading to google: please take a look at this good answer: How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?) and follow those guidelines.

A nice way to go with GAE is through yaml application directives. Please take a look at the doc for includes: http://code.google.com/appengine/docs/python/config/appconfig.html#Includes

Also remember that GAE officially supports python 2.5, and 2.7 support is experimental

Python 2.7 is now officially supported

Answer 2

To properly import a module, you need to make sure, that python knows where to find it. To do so, for each external library append it's parent directory to the sys.path (in run-time), or setup PYTHONPATH environment (before running).

For example:

import sys
sys.path.append('/my/lib')

# now we can import from lib
import jsonpickle # will load /my/lib/jsonpickle/__init__.py

See http://docs.python.org/tutorial/modules.html#the-module-search-path . to understand what python does when you call import.