Celery: Task Singleton?

Question

I have a task that I need to run asynchronously from the web page that triggered it. This task runs rather long, and as the web page could be getting a lot of these requests, I'd like celery to only run one instance of this task at a given time.

Is there any way I can do this in Celery natively? I'm tempted to create a database table that holds this state for all the tasks to communicate with, but it feels hacky.

Answer 1

You probably can create a dedicated worker for that task configured with CELERYD_CONCURRENCY=1 then all tasks on that worker will run synchronously

Answer 2

You can use memcache/redis for that. There is an example on the celery official site - http://docs.celeryproject.org/en/latest/tutorials/task-cookbook.html

And if you prefer redis (This is a Django realization, but you can also easily modify it for your needs):

from django.core.cache import cache
from celery.utils.log import get_task_logger


logger = get_task_logger(__name__)


class SingletonTask(Task):
    def __call__(self, *args, **kwargs):
        lock = cache.lock(self.name)

        if not lock.acquire(blocking=False):
            logger.info("{} failed to lock".format(self.name))
            return

        try:
            super(SingletonTask, self).__call__(*args, **kwargs)
        except Exception as e:
            lock.release()
            raise e
        lock.release()

And then use it as a base task:

@shared_task(base=SingletonTask)
def test_task():
    from time import sleep
    sleep(10)

This realization is nonblocking. If you want next task to wait for the previous task change blocking=False to blocking=True and add timeout