How does caching affect memory consumption?

Question

I have an app which has a search feature. This feature looks up the search term in a giant object (dictionary) that I cache for 24 hours. The object is about 50,000 keys and weighs roughly 10MB.

When I profile the memory usage on my hosting, I notice that after a few queries, the memory usage goes from around 50MB to over 450MB, prompting my hosting provider to kill the app.

So I'm wondering what is going on here. Specifically, how does the cache utilize the memory on each request and what can I do to fix this?

Answer 1

Django FileBasedCache is known for having performance issues. You can get a big picture on the following links:

A smarter filebasedcache for Django

Bug: File based cache not very efficient with large amounts of cached files

Bug was set as wont fix arguing:

I'm going to wontfix, on the grounds that the filesystem cache is intended as an easy way to test caching, not as a serious caching strategy. The default cache size and the cull strategy implemented by the file cache should make that obvious.