How to limit the maximum number of running Celery tasks by name

Question

How do you limit the number of instances of a specific Celery task that can be ran simultaneously?

I have a task that processes large files. I'm running into a problem where a user may launch several tasks, causing the server to run out of CPU and memory as it tries to process too many files at once. I want to ensure that only N instances of this one type of task are ran at any given time, and that other tasks will sit queued in the scheduler until the others complete.

I see there's a rate_limit option in the task decorator, but I don't think this does what I want. If I'm understanding the docs correctly, this will just limit how quickly the tasks are launched, but it won't restrict the overall number of tasks running, so this will make my server will crash more slowly...but it will still crash nonetheless.

Answer 1

You have to setup extra queue and set desired concurrency level for it. From Routing Tasks:

# Old config style    
CELERY_ROUTES = {
                'app.tasks.limited_task': {'queue': 'limited_queue'}
            } 

or

from kombu import Exchange, Queue
celery.conf.task_queues = (
        Queue('default', default_exchange, routing_key='default'),
        Queue('limited_queue', default_exchange, routing_key='limited_queue')
    ) 

And start extra worker, serving only limited_queue:

$ celery -A celery_app worker -Q limited_queue --loglevel=info -c 1 -n limited_queue

Then you can check everything running smoothly using Flower or inspect command:

$ celery -A celery_app worker inspect --help

Answer 2

What you can do is to push these tasks to a specific queue and have X number of workers processing them. Having two workers on a queue with 100 items will ensure that there will only be two tasks processed at the same time.