TemplateDoesNotExist on python app-engine django 1.2 while template rendering relative paths

Question

I'm running the 1.4.2 appengine SDK locally on a windows machine. I have an application running Django 0.96. The template rendering is using the django wrapper from

google.appengine.ext.webapp.template.render

to render templates. I often use a relative path to link my templates e.g

{% extends "../templates/base.html" %}

After upgrading to Django 1.2 the find_template method from django.template.loader in the appengine's Django 1.2 lib folder is now raising a TemplateDoesNotExist when the relative paths are used

for loader in template_source_loaders:
    try:
        #raises TemplateDoesNotExist name='../templates/home.html' dirs=None
        source, display_name = loader(name, dirs)
        return (source, make_origin(display_name, loader, name, dirs))
    except TemplateDoesNotExist:
        pass
raise TemplateDoesNotExist(name)

I've been stepping through the Django and AppEngine code for a while now but can't see any reason for this. Can anyone provide any more insight?

Thanks,

Richard

Answer 1

This problem bit me too when I converted from 0.96 to 1.2 Django templates. I was initially pushed to do so when SDK 1.4.2 started issuing the warning that I needed to pick a version, but when I looked into the much-needed improvements in the template language, I was eager to make the change.

And then everything broke. Like you, I used a lot of relative paths in my extends and include commands. It took a lot of debugging and digging, but I did figure out the cause of the problem and a pretty good solution.

The cause: in Django 1.2, the code that loads template files started using a command called safe_join to join path parts (you can see the code in google_appengine\lib\django_1_2\django\template\loaders\filesystem.py) . It won't allow relative paths to go above what it thinks of as the top-level directory. This is the same thing as a web server being configured to prevent you gaining access to the server's whole filesystem just by sticking some ..'s into your URL. The end result is that the

{% extends "../templates/base.html" %}

that used to be just fine breaks the rules and it isn't going to work.

The way that I fixed this in my application without completely restructuring how my templates are laid out is by implementing a custom TemplateLoader. Django's template rendering engine allows an application to have many different classes that know how to find templates in different ways. If you look in the directory that I gave above, you'll see that there are several provided, and they are all classes that inherit from BaseLoader. I provided my own that is custom-tailored to how my templates are laid out.

My project has a Rails-like lay-out:

app/
   controllers/
      home_controller.py
      posts_controller.py
   models/
      ...
   views/
      home/
          index.html
          about.html
      posts/
          show.html
          new.html
      shared/
          base.html
          post.html

Every template extends base.html and a couple include post.html, and they previously used relative paths to get to their location in base/. Ideally, I didn't even want to use the .. up-dir to get there, but it was required with 0.96. I created the following template loader to work with my scheme:

from django.conf import settings
from django.template import TemplateDoesNotExist
from django.template.loader import BaseLoader
from django.utils._os import safe_join
import os

class MvcTemplateLoader(BaseLoader):
    "A custom template loader for the MVCEngine framework."

    is_usable = True

    __view_paths = None

    def __init__(self, views_path):
        self.views_path = views_path
        # We only need to instantiate the view_paths class variable once.
        if MvcTemplateLoader.__view_paths is None:
            temp_paths = []
            for each_path in os.listdir(views_path):
                # We want to skip hidden directories, so avoid anything that starts with .
                # This works on both Windows and *NIX, but could it fail for other OS's?
                if not each_path.startswith('.'):
                    full_path = os.path.join(views_path, each_path)
                    if each_path == "shared":
                        # The shared directory is special. Since templates in many other directories will be
                        # inheriting from or including templates there, it should come second, right after the
                        # root views directory. For now, it will be first.
                        temp_paths.insert(0, full_path)
                    else:
                        temp_paths.append(full_path)
            # The root views_path itself will always be first in order to give resolution precendence to templates
            # that are specified with a parent directory. In other words, home/index.html will be immediately
            # resolved with no ambiguity; whereas, index.html could resolve as bar/index.html rather than
            # foo/index.html.
            temp_paths.insert(0, views_path)
            MvcTemplateLoader.__view_paths = temp_paths


    def get_template_sources(self, template_name):
        for template_dir in MvcTemplateLoader.__view_paths:
            try:
                yield safe_join(template_dir, template_name)
            except UnicodeDecodeError:
                # The template dir name was a bytestring that wasn't valid UTF-8.
                raise
            except ValueError:
                # The joined path was located outside of this particular
                # template_dir (it might be inside another one, so this isn't
                # fatal).
                pass

    def load_template_source(self, template_name, template_dirs=None):
        tried = []
        for filepath in self.get_template_sources(template_name):
            try:
                file = open(filepath)
                try:
                    return (file.read().decode(settings.FILE_CHARSET), filepath)
                finally:
                    file.close()
            except IOError:
                tried.append(filepath)

        error_msg = "Could not find %s in any of the views subdirectories." % template_name
        raise TemplateDoesNotExist(error_msg)
    load_template_source.is_usable = True

_loader = MvcTemplateLoader

And I caused my custom template loader to be included in the collection that Django tries by changing my application's main function to look like this:

def main():    
    from google.appengine.dist import use_library
    use_library('django', '1.2')

    os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

    from django.conf import settings 
    views_path = os.path.join(os.path.dirname(__file__), 'app','views')
    settings.TEMPLATE_LOADERS = (('gaemvclib.mvctemplateloader.MvcTemplateLoader', views_path), 'django.template.loaders.filesystem.Loader', 'django.template.loaders.app_directories.Loader')

(and then all the rest of the stuff that normally goes into your main function).

So, I think that you should be able to modify the TemplateLoader code above to match how you have your template directories laid out, and it will give you a greater control over not only how you layout you templates hierarcy but also how you write your extends and include statement. You no longer use .. but rather just give the path of the template relative to whatever in your project is the equivalent of my views directory.