Is Django filesystem cache thread-safe? and is it cross-process?

Question

I need to cache lists 10MB+ size, is it safe to do it in Django filesystem cache?

Or is there a better way to do it?

Answer 1

I would examine carefully the Python DiskCache.

Django is Python’s most popular web framework and ships with several caching backends. Unfortunately the file-based cache in Django is essentially broken. The culling method is random and large caches repeatedly scan a cache directory which slows linearly with growth. Can you really allow it to take sixty milliseconds to store a key in a cache with a thousand items?

In Python, we can do better. And we can do it in pure-Python!