Hierarchical cache in Django

Question

What I want to do is to mark some values in the cache as related so I could delete them at once. For example when I insert a new entry to the database I want to delete everything in the cache which was based on the old values in database.

I could always use cache.clear() but it seems too brutal to me. Or I could store related values together in the dictionary and cache this dictionary. Or I could maintain some kind of index in an extra field in cache. But everything seems to complicated to me (eventually slow?).

What you think? Is there any existing solution? Or is my approach wrong? Thanks for answers.

Answer 1

Are you using the cache api? It sounds like it.

This post, which pointed me to these slides helped me create a nice generational caching system which let me create the hierarchy I wanted.

In short, you store a generation key (such as group) in your cache and incorporate the value stored into your key creation function so that you can invalidate a whole set of keys at once.

With this basic concept you could create highly complex hierarchies or just a simple group system.

For example:

class Cache(object):
    def generate_cache_key(self, key, group=None):
        """
        Generate a cache key relating them via an outside source (group)
        Generates key such as 'group-1:KEY-your-key-here'

        Note: consider this pseudo code and definitely incomplete code.
        """
        key_fragments = [('key', key)]

        if group:
            key_fragments.append((group, cache.get(group, '1')))

        combined_key = ":".join(['%s-%s' % (name, value) for name, value in key_fragments)

        hashed_key = md5(combined_key).hexdigest()
        return hashed_key


    def increment_group(self, group):
        """
        Invalidate an entire group
        """
        cache.incr(group)


    def set(self, key, value, group=None):
        key = self.generate_cache_key(key, group)
        cache.set(key, value)


    def get(self, key, group=None):
        key = self.generate_cache_key(key, group)
        return cache.get(key)

# example
>>> cache = Cache()
>>> cache.set('key', 'value', 'somehow_related')
>>> cache.set('key2', 'value2', 'somehow_related')
>>> cache.increment_group('somehow_related')
>>> cache.get('key') # both invalidated
>>> cache.get('key2') # both invalidated