AppEngine application using Django fails to load

Question

Django is constantly causing our application to crash. After deployment the application is running fine, but once the initial instance is restarted/shutdown it often fails to start with an error similar to the following:

Traceback (most recent call last):  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
    result = handler(dict(self._environ), self._StartResponse)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/wsgi.py", line 236, in call
    self.load_middleware()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/base.py", line 53, in load_middleware
    raise exceptions.ImproperlyConfigured('Error importing middleware %s: "%s"' % (mw_module, e))
ImproperlyConfigured: Error importing middleware myfolder.middleware: "No module named myfolder.middleware"

Our file structure is similar to this:

|- app.yaml
|- _ _ init _ _.py
|- settings.py
|- myfolder |
|           |- _ _ init _ _.py
|           |- middleware.py
|           |- ...
|-...
|

Our app.yaml:

application: XXXXX
module: app
version: master
runtime: python27
api_version: 1
threadsafe: true

handlers:
- url: /api/(login|logout|passwd|master.|banners.)
  script: app.handler
  secure: always
...

builtins:
- django_wsgi: on

libraries:
- name: django
  version: 1.5

env_variables:
    DJANGO_SETTINGS_MODULE: 'settings'

We have 2 modules in our application and they both exhibit this behaviour (they have similar configurations). Sometimes the modules will stay up for a whole day before crashing again. After they fail to load, all subsequent requests fail with he same error. Deploying one more time always solves the problem temporarily.

We are using plain django with CloudSql. The problem is not reproducible in the development server. After deployment everything in both modules works fine. All middleware, ndb, memcache, cloudsql, taskqueue, etc, including all the modules inside the "myfolder" and every other library xcopied.

The following attempts at solving this problem haven't worked:

• We have tried using the appengine_config.py to force django to reload the settings with from django.conf import settings\nsettings._target = None\n
• Originally we had shared settings inside "myfolder" and were importing them with "from myfolder.shared_settings import *" inside the root settings.py but django could not load the module myfolder.shared_settings either (similar problem)
• using a custom mysettings.py and defining the DJANGO_SETTINGS_MODULE in the app.yaml or in python

The system is not live yet but will be soon and we are running out of options.

Other traces of similarly failing configurations:

Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 266, in Handle
    result = handler(dict(self._environ), self._StartResponse)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/wsgi.py", line 236, in __call__
    self.load_middleware()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/core/handlers/base.py", line 45, in load_middleware
    for middleware_path in settings.MIDDLEWARE_CLASSES:
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 53, in __getattr__
    self._setup(name)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 48, in _setup
    self._wrapped = Settings(settings_module)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 134, in __init__
    raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'settings' (Is it on sys.path?): No module named myfolder.settings

Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 239, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/lib_config.py", line 353, in __getattr__
    self._update_configs()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/lib_config.py", line 289, in _update_configs
    self._registry.initialize()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/lib_config.py", line 164, in initialize
    import_func(self._modname)
  File "/base/data/home/apps/s~blue-myapp/app:master.375531077560785947/appengine_config.py", line 17, in 
    settings._target = None
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/utils/functional.py", line 227, in __setattr__
    self._setup()
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 48, in _setup
    self._wrapped = Settings(settings_module)
  File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5/django/conf/__init__.py", line 134, in __init__
    raise ImportError("Could not import settings '%s' (Is it on sys.path?): %s" % (self.SETTINGS_MODULE, e))
ImportError: Could not import settings 'settings' (Is it on sys.path?): No module named myfolder.settings

This is our current appengine_config.py:

import sys
import logging

logging.debug(",\n".join(sys.path))

# Since Google App Engine's webapp framework uses Django templates, Django will half-initialize when webapp is loaded.
# This causes the initialization of the rest of Django's setting to be skipped. If you are getting this error, you need
# to explicitly force Django to reload your settings:

from django.conf import settings
settings._target = None

Logging sys.path from appengine_config.py does not change between a successful instance start and a failed instance start (apart from the XXXXXXXXXXX bit of course):

/base/data/home/apps/s~blue-persomi/app:master.3759720XXXXXXXXXXX,
/base/data/home/runtimes/python27/python27_dist/lib/python27.zip,
/base/data/home/runtimes/python27/python27_dist/lib/python2.7,
/base/data/home/runtimes/python27/python27_dist/lib/python2.7/plat-linux2,
/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-tk,
/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-old,
/base/data/home/runtimes/python27/python27_dist/lib/python2.7/lib-dynload,
/base/data/home/runtimes/python27/python27_dist/lib/python2.7/site-packages,
/base/data/home/runtimes/python27/python27_lib/versions/1,
/base/data/home/runtimes/python27/python27_lib/versions/third_party/MySQLdb-1.2.4b4,
/base/data/home/runtimes/python27/python27_lib/versions/third_party/django-1.5,
/base/data/home/runtimes/python27/python27_lib/versions/third_party/protorpc-1.0,
/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2,
/base/data/home/runtimes/python27/python27_lib/versions/third_party/webob-1.1.1,
/base/data/home/runtimes/python27/python27_lib/versions/third_party/yaml-3.10

Answer 1

It seems to be a path related issue as people have mentioned in your question's comments

Possible short-sighted solution add everything to your path manually - look at the top answer here: How to import modules in Google App Engine? At the very least, this will help narrow the problem to path related.

what the docs say: https://developers.google.com/appengine/docs/python/

The Python module include path includes your application's root directory (the directory containing the app.yaml file). Modules you create in your application's root directory are available using a path from the root. Don't forget to create init.py files in sub-directories, so Python will recognize the sub-directories as packages.

so from what I can tell, b/c everything is at or below the app.yaml file in your question the path should already be correct.

1. double check all your __init__.py file files are in place and spelled correctly.
2. try deleting all of your *.pyc files and letting them be regenerated.
3. try importing from the container's folder name FOLDER_CONTAINING_YAML.myfolder.middleware