How to cleanup celery_taskmeta?

Question

I've been reading django-celery to try to find a native way of cleaning up celery_taskmeta, so this table won't grow up too much when in production environment.

Can'f find nothing. Should i do a command to deal with this?

Answer 1

I'm not using django-celery so my answer could be incomplete.

There is a method in the Backend that allow you to force cleanup of celery_taskmeta : in celery.backends.base.BaseBackend, you can find method cleanup()

So, when I need to force cleanup I have 3 ways :

1. using the AsyncResult on producer side
2. using the task on the consumer side
3. celerybeat task

on producer side

# Launch a task
my_task = add_a_task() # use the Task.delay() method for example
# my_task is a celery.result.AsyncResult instance
# Retrieve backend and call cleanup
my_task.backend.cleanup()

on the consumer side

from celery import Task
from .celery_app import my_celery_app

# Class for a task
@my_celery_app.task()
class CopyTask(Task):
    # A Task instance got a backend property

    def run(self,**kwargs):
        """ Running method of the task """
        # Do something
        pass

    def force_cleanup(self):
        """ Force cleanup. """
       self.backend.cleanup()

Note that calling a cleanup in the task itself seems quiet awfull.

with celerybeat task

A third option, is using a celerybeat task that cleanup backend celery_taskmeta. This is the one I probably choose.

celery beat is basically a scheduler and you can run the task you want. So you can have a dedicated task, like this one :

from celery import Task
from .celery_app import my_celery_app

# Class for a task
@my_celery_app.task()
class CleanupTask(Task):
    # A Task instance got a backend property

    def run(self,**kwargs):
        """ Running method of the task """
        # Cleanup here
        self.backend.cleanup()

Setting a celery beat, like a crontab job is well documented

Answer 2

I have a cronjob running a Django command that executes this SQL statement:

"truncate table celery_taskmeta"

deleted 50.000 records in 0.47ms.