Django caching a large list

Question

My django application deals with 25MB binary files. Each of them has about 100,000 "records" of 256 bytes each.

It takes me about 7 seconds to read the binary file from disk and decode it using python's struct module. I turn the data into a list of about 100,000 items, where each item is a dictionary with values of various types (float, string, etc.).

My django views need to search through this list. Clearly 7 seconds is too long.

I've tried using django's low-level caching API to cache the whole list, but that won't work because there's a maximum size limit of 1MB for any single cached item. I've tried caching the 100,000 list items individually, but that takes a lot more than 7 seconds - most of the time is spent unpickling the items.

Is there a convenient way to store a large list in memory between requests? Can you think of another way to cache the object for use by my django app?

Answer 1

edit the item size limit to be 10m (larger than 1m), add

-I 10m

to /etc/memcached.conf and restart memcached

also edit this class in memcached.py located in /usr/lib/python2.7/dist-packages/django/core/cache/backends to look like this:

class MemcachedCache(BaseMemcachedCache):
"An implementation of a cache binding using python-memcached"
def __init__(self, server, params):
    import memcache
    memcache.SERVER_MAX_VALUE_LENGTH = 1024*1024*10 #added limit to accept 10mb
    super(MemcachedCache, self).__init__(server, params,
                                         library=memcache,
                                         value_not_found_exception=ValueError)