Django cache.clear() not working (LocMemCache, AWS)

Question

Background

I have a website running Django 2.0 on AWS ElasticBeanstalk. I have a couple views on my website that take some time to calculate, so I thought I'd look into some simple caching. I decided on LocMemCache because it looked like the quickest to set up that would meet my needs. (I'm using AWS, so using Memcached apparently requires ElastiCache, which adds cost and is additional setup overhead that I wanted to avoid.)

The views do not change often, and the site is not high-traffic, so I put long timeouts on the caches. There are three views where I have enabled caching:

1. A report generated inside a template – uses Template Fragment caching
2. A list of locations requested by AJAX and used in a JS library – uses per-view caching
3. A dynamically-generated binary file download – uses per-view caching

The caching is set up and works great.

The data that goes into these views is added and edited by other staff at my company, that are used to their changes appearing immediately. So in order to address questions such as, "I updated this data, why has the webpage not updated?" I wanted to create a "Clear Server Cache" button, accessible by staff, to force a cache reset.

The button is set up and functioning. It requests a view that calls cache.clear() from django.core.cache. I used the sledgehammer cache.clear() approach because the way to specify an individual per-view cache in code seems to be a bit clunky and convoluted, so the "clear it all" approach seemed adequate. And at the very least it should always "work" in the sense that all the data will get re-loaded again.

The Problem

When I use the button to call cache.clear(), it only clears the Template Fragment cache. It does not seem to clear the per-view caches. Why?

According to Django Documentation,

Be careful with this; clear() will remove everything from the cache, not just the keys set by your application.

So why is it not touching the per-view caches? Doesn't the warning seem to indicate that clear() is dangerous specifically because it's a sledgehammer and nothing at all is spared? What am I missing?

Does AWS use some kind of special memory that's immune to this sort of culling? (If this is the case, then why are the Template Fragments successfully cleared?) I did notice (and find it interesting) that the cache remains even after deploying a new image to the same environment.

I could switch to using Database caching, but I'd like to understand why this isn't working so I don't need to abandon LocMemCache as an option to ever use in the future.

I could also move the others to use Template Fragment caching, but if I ever expand the caching to fit other needs, I will want to be able to use per-view caching. Also, this solution would be less than ideal for a binary-file-download view.

settings.py

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
##        'LOCATION': '',
    },
}

portfolio.html (Cache #1 – Template Fragment)

{% load static cache compress %}

...

<div id="total-portfolio-content">
{% cache 7200 portfolio %}{% include 'reports/total_portfolio/report_include.html' %}{% endcache %}
</div>

map/urls.py (Cache #2 – Per-view)

from django.conf.urls import include, url

app_name = 'map'
urlpatterns = [
    ## Yes, I know this uses the old-style url(). I have plans to upgrade the entire project.
    url(r'^(?P<tg>[\w-]+)/data.geojson$',
        cache_page(60 * 60 * 12)(
            views.NamedGeoJSONLayerView.as_view(model=FacilityCoord)),
        name='tg-data'),
]

resources/urls/__init__.py (Cache #3 – Per-view)

from django.conf.urls import include, url

app_name = 'resources'
urlpatterns = [
    url(r'^download/$',
        cache_page(60 * 60 * 12)(
            views.DownloadMetricXLSX.as_view()),
        name='download'),
]

myadmin/views.py (Cache Clear button)

from django.core.cache import cache

@staff_member_required(login_url=login_url)
def clear_cache(request):
    cache.clear()

    ## And because that doesn't seem to work as advertised, I also tried....
    ## taken from <https://djangosnippets.org/snippets/1080/>
    try:
        cache._cache.clear()  # in-memory caching
        cache._expire_info.clear()
    except AttributeError:
        # I think this only applies to filesystem caching? Just grasping at straws.
        old_freq = cache._cull_frequency
        old_max = cache._max_entries
        cache._max_entries = 0
        cache._cull_frequency = 1
        cache._cull()
        cache._cull_frequency = old_freq
        cache._max_entries = old_max

    return JsonResponse({'success': True})

Answer 1

The problem is in the response headers. The cache_page decorator automatically adds a max-age option to the Cache-Control header in the response. So the cache clear was working properly, clearing the local memory on the server, but the user's browser was instructed not to ask the server for updated data for the duration of the timeout. And my browser was happily complying (even after Ctrl-F5).

Fortunately, there are other decorators you can use to deal with this without much difficulty, now that it's clear what's happening. Django provides a number of other decorators, such as cache_control or never_cache.

I ended up using never_cache, which turned the urls files into...

from django.conf.urls import include, url
from django.views.decorators.cache import never_cache, cache_page

app_name = 'map'
urlpatterns = [
    url(r'^(?P<tg>[\w-]+)/data.geojson$',
        never_cache(cache_page(60 * 60 * 12)(
            views.NamedGeoJSONLayerView.as_view(model=FacilityCoord))),
        name='tg-data'),
]

and

from django.conf.urls import include, url
from django.views.decorators.cache import never_cache, cache_page

app_name = 'resources'
urlpatterns = [
    url(r'^download/$',
        never_cache(cache_page(60 * 60 * 12)(
            views.DownloadMetricXLSX.as_view())),
        name='download'),
]