Django 1.6 and Celery 3.0 memory leaks

Question

After upgrading Django to 1.6, my celery worker is eating up RAM. Seems that the memory allocated for the workers isn't released and grows after every task.

Related Settings:

# DB:
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'somedb',
        'USER': '',
        'PASSWORD': '',
        'HOST': 'localhost',
        'PORT': '',
    }
}


# CELERY SETTINGS:
CELERY_RESULT_BACKEND = 'redis://'
BROKER_URL = 'redis://'

Related package versions:

Django==1.6
celery==3.0.24
django-celery==3.0.23
billiard==2.7.3.34
kombu==2.5.16
redis==2.7.6

Happens in both my local env (with DEBUG=False) running the worker manually and in a staging environment where celery is running with Upstart.

---

Updates:

1. Tried setting autocommit=False with no success.
2. Could be it's not related to the Django version upgrade, but to some setting or 3rd party package that I had to upgrade to make the switch to 1.6.

Answer 1

It turns out the memory leak was not directly caused by the Django upgrade or Celery.

After a lot of digging around I found that, surprisingly, the celery worker memory leak happens because I upgraded django-debug-toolbar from 0.9.4 to 0.11.0 (which is needed for Django 1.6 compatibility).

Still no idea what exactly caused this issue, or why it only happens in the celery worker processes and not in the app server ones (Gunicorn).

Removing django-debug-toolbar from the installed apps and middleware solves the issue. At least temporarily.

Answer 2

It appears that the change from django-debug-toolbar 0.9.4 to 0.11.0 did introduce a memory leak caused by the LoggingPanel storing an infinite number of messages. If you had a process that was using the logging subsystem, it's likely you ran into this problem. You can also remove the LoggingPanel from the list of default panels to work around the problem.

Apparently in 0.9.4, the panels were lazily initialized only when the middleware was accessed. This changed in 0.11.0: the panels are initialized immediately after import, and the LoggingPanel module was intercepting all logs regardless whether DEBUG was set.

I've submitted a fix for this bug.