TypeError from Django Cache

Question

Update: See "Even Better Solution" below.

This one has me stumped. When I call an HTML page generated by Django REST Framework, it works. When I call it a second, third, fourth time, I get:

[26/Oct/2015 15:14:42]"GET /api/rest/v3/dockets/ HTTP/1.1" 500 92424
Internal Server Error: /api/rest/v3/dockets/
Traceback (most recent call last):
  File "/home/mlissner/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 108, in get_response
    response = middleware_method(request)
  File "/home/mlissner/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/django/middleware/cache.py", line 134, in process_request
    response = self.cache.get(cache_key, None)
  File "/home/mlissner/.virtualenvs/courtlistener/local/lib/python2.7/site-packages/django/core/cache/backends/locmem.py", line 54, in get
    return pickle.loads(pickled)
TypeError: __new__() takes exactly 3 arguments (2 given)

Unlike 99% of the stacktraces I get from Django, this one doesn't mention my code at all, and seems to be only code from Django itself.

I'm using the development server, Django 1.8.7, Django REST Framework 3.2.3, and Python 2.7.

My middleware setting is:

MIDDLEWARE_CLASSES = [
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.contrib.auth.middleware.SessionAuthenticationMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.middleware.cache.UpdateCacheMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'corsheaders.middleware.CorsMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware',
]

I've only seen this when looking at Django REST Framework pages. Any ideas?

Things I've Tried

• Upgrading Django, djangorestframework and djangorestframework-filters.
• Changing my CACHE setting so it uses Redis instead of LocMemCache. I thought this might help because somebody in the comments said changing it to FileBasedCache helped. The change to redis, in itself did not help, though setting it to the DummyCache does.

Solution

django-redis-cache allows you to set different versions of pickle so I've tinkered with that, given that one link hinted at the pickle version being related. At first, this seemed to have no effect, but I filed a bug in django-redis ("PICKLE_VERSION doesn't seem to work"), which they quickly fixed. Once that was fixed, I set the PICKLE_VERSION to 1, and the problem was solved.

I've also filed a bug in DRF to see if there's a better way to get to the bottom of this. However, I'm not sure if the bug is there, in my code, or in Django itself.

Even better solution

I'm the master at workarounds, it seems. But the good news is that this was a bug in Django Rest Framework, which has been fixed, and will be released in 3.3.2 (hopefully).

Answer 1

It is easy to understand why your problem only happens when accessing the page after the second time. That is because when you load the page the first time, the data is fetched from data source (database?), and written into cache. After that, each page load will hit the cache directly.

It seems the problem is related to what type of data is cached. The exception is raised in pickle module, when the cached data is loaded from a string, the unpickler detects the type of data incorrectly, and call __new__ method of that class. That is where the error occurs.

There are some posts talking about the pickle unload error, see if they are helpful:

• http://bugs.python.org/issue3065

• http://www.gossamer-threads.com/lists/python/python/373825

It looks there were issues when pickling a tuple, does the data cached in your app include a tuple?