RabbitMQ/Celery/Django Memory Leak?

Question

I recently took over another part of the project that my company is working on and have discovered what seems to be a memory leak in our RabbitMQ/Celery setup.

Our system has 2Gb of memory, with roughly 1.8Gb free at any given time. We have multiple tasks that crunch large amounts of data and add them to our database.

When these tasks run, they consume a rather large amount of memory, quickly plummeting our available memory to anywhere between 16Mb and 300Mb. The problem is, after these tasks finish, the memory does not come back.

We're using:

• RabbitMQ v2.7.1
• AMQP 0-9-1 / 0-9 / 0-8 (got this line from the RabbitMQ startup_log)
• Celery 2.4.6
• Django 1.3.1
• amqplib 1.0.2
• django-celery 2.4.2
• kombu 2.1.0
• Python 2.6.6
• erlang 5.8

Our server is running Debian 6.0.4.

I am new to this setup, so if there is any other information you need that could help me determine where this problem is coming from, please let me know.

All tasks have return values, all tasks have ignore_result=True, CELERY_IGNORE_RESULT is set to True.

Thank you very much for your time.

My current config file is:

CELERY_TASK_RESULT_EXPIRES = 30
CELERY_MAX_CACHED_RESULTS = 1
CELERY_RESULT_BACKEND = False
CELERY_IGNORE_RESULT = True
BROKER_HOST = 'localhost'
BROKER_PORT = 5672
BROKER_USER = c.celery.u
BROKER_PASSWORD = c.celery.p
BROKER_VHOST = c.celery.vhost

Answer 1

I am almost certain you are running this setup with DEBUG=True wich leads to a memory leak.

Check this post: Disable Django Debugging for Celery.

I'll post my configuration in case it helps.

settings.py

djcelery.setup_loader()
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_VHOST = "rabbit"
BROKER_USER = "YYYYYY"
BROKER_PASSWORD = "XXXXXXX"

CELERY_IGNORE_RESULT = True
CELERY_DISABLE_RATE_LIMITS = True
CELERY_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1
CELERYBEAT_SCHEDULER = "djcelery.schedulers.DatabaseScheduler"
CELERY_ROUTES = ('FILE_WITH_ROUTES',)

Answer 2

You might be hitting this issue in librabbitmq. Please check whether or not Celery is using librabbitmq>=1.0.1.

A simple fix to try is: pip install librabbitmq>=1.0.1.