Patterns for dealing with memcache Caching in Django

Question

I have a big Django project with several interrelated projects and a lot of caching in use. It currently has a file which stores cache helper functions. So for example, get_object_x(id) would check cache for this object and if it wasn't there, go to the DB and pull it from there and return it, caching it along the way. This same pattern is followed for caching groups of objects and the file is also used for for invalidation methods.

A problem has arisen in the imports between apps though. The app models file has a number of helper functions which we want to use cache for, and the cache_helpers file obviously needs to import the models file.

So my question is: What is a better way of doing this that doesn't expose the code to circular import issues (or maybe just a smarter way in general)? Ideally we could do invalidation in a better, less manual way as well. My guess is that the use of Django Custom Managers and Signals is the best place to start, getting rid of the cache_helpers file altogether, but does anyone have any better suggestions or direction on where to look?

Answer 1

A general Python pattern for avoiding circular imports is to put one set of the imports inside the dependent functions:

# module_a.py
import module_b

def foo():
    return "bar"

def bar():
    return module_b.baz()

# module_b.py
def baz():
    import module_a
    return module_a.foo()

As for caching, it sounds like you need a function that looks a bit like this:

def get_cached(model, **kwargs):
    timeout = kwargs.pop('timeout', 60 * 60)
    key = '%s:%s' % (model, kwargs)
    result = cache.get(key)
    if result is None:
        result = model.objects.get(**kwargs)
        cache.set(key, result, timeout)
    return result

Now you don't need to create "getbyid" methods for every one of your models. You can do this instead:

blog_entry = get_cached(BlogEntry, pk = 4)

You could write similar functions for dealing with full QuerySets instead of just single model objects using .get() method.

Answer 2

Since you indicated you're caching Django ORM model instances, take a look at django-orm-cache, which provides automated caching of model instances and is smart about when to invalidate the cache.

Your circular imports won't be an issue - all you need to do is extend the models you need to cache from the ormcache.models.CachedModel class instead of Django's django.db.models.Model, and you get caching "for free."