Why doesn't memory get released to system after large queries (or series of queries) in django?

Question

First off, DEBUG = False in settings.py, so no, connections['default'].queries is not growing and growing until it uses up all of memory.

Lets start off with the fact that I've loaded the User table from django.contrib.auth.models.User with 10000 users (each named 'test#' where # is a number between 1 and 10000).

Here is the view:

from django.contrib.auth.models import User
from django.http import HttpResponse

import time

def leak(request):
    print "loading users"

    users = []
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())
    users += list(User.objects.all())

    print "sleeping"
    time.sleep(10)

    return HttpResponse('')

I've attached the view above to the /leak/ url and start the development server (with DEBUG=False, and I've tested and it has nothing to do with running a development server vs other instances).

After running:

% curl http://localhost:8000/leak/

The runserver process' memory grows to around the size seen from ps aux output below and then stays at that level.

USER       PID %CPU %MEM    VSZ    RSS TTY      STAT START   TIME COMMAND
dlamotte 25694 11.5 34.8 861384 705668 pts/3    Sl+  19:11   2:52 /home/dlamotte/tmp/django-mem-leak/env/bin/python ./manage.py runserver

Then running the above curl command above does not seem to grow the instance's memory usage (which I expected from a true memory leak?), it must be re-using the memory? However, I feel that there is something wrong here that the memory does not get released to the system (however, I understand that it may be better performance that python does NOT release the memory).

Following this, I naively attempted to see if python would release large chunks of memory that it allocated. So I attempt the following from a python session:

>>> a = ''
>>> a += 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' * 10000000
>>> del a

The memory is allocated on the a += ... line as expected, but when del a happens, the memory is released. Why is the behavior different for django query sets? Is it something that django is intending to do? Is there a way to change this behavior?

I've literally spent 2 days debugging this behavior with no idea where to go next (I've learned to use guppy AND objgraph which seem to not point to anything interesting that I can figure out).

UPDATE: This could be simply python memory management at work and have nothing to do with Django (suggested on django-users mailing list), but I'd like confirmation by somehow replicating this in python outside of Django.

UPDATE: Using python version 2.6.5

Answer 1

I decided to move my comments into an answer to make things clearer.

Since Python 2.5, the CPython memory allocation tracks internal memory usage by the small object allocator, and attempts to return completely free arenas to the underlying OS. This works most of the time, but the fact that objects can't be moved around in memory means that fragmentation can be a serious problem.

Try the following experiment (I used 3.2, but 2.5+ should be similar if you use xrange):

# Create the big lists in advance to avoid skewing the memory counts
seq1 = [None] * 10**6 # Big list of references to None
seq2 = seq1[::10]

# Create and reference a lot of smaller lists
seq1[:] = [[] for x in range(10**6)] # References all the new lists
seq2[:] = seq1[::10] # Grab a second reference to 10% of the new lists

# Memory fragmentation in action
seq1[:] = [None] * 10**6 # 90% of the lists are no longer referenced here
seq2[:] = seq1[::10] # But memory freed only after last 10% are dropped

Note, even if you drop the references to seq1 and seq2, the above sequence will likely leave your Python process holding a lot of extra memory.

When people talk about PyPy using less memory than CPython, this is a major part of what they're talking about. Because PyPy doesn't use direct pointer references under the hood, it is able to use a compacting GC, thus avoiding much of the fragmentation problem and more reliably returning memory to the OS.

Answer 2

Lots of applications, language runtimes, and perhaps even some system memory allocators will keep deallocated memory in place for as long as possible with a view to re-using it, purely for performance purposes. In a complex system like Django it could be any number of extensions, possibly implemented in C, which exhibit this behaviour, or it might be Python with some sort of memory pool, or lazy garbage collection.

It could even be the underlying malloc implementation doing this, or your operating system keeping a certain amount of memory space allocated to your process even though the process isn't explicitly using it. Don't quote me on this though - it's been a while since I looked into such things.

On the whole though, if repeating the allocation process after the initial alloc and dealloc does not double the amount of memory used, what you're seeing is not a memory leak but memory pooling. It's only likely to be a problem if you have a lot of processes that contend for a limited amount of memory on that machine.