Run celery periodic task for 1 hour, it it takes more then 1 hour, expire that task?

Question

I have a periodic task for data purging which run once a day.

I want that task to run only for 1 hour. If the duration of processing that task is more than 1 hour, then expire that task. It will run again the next day for 1 hour.

I would like to do this is because if the traffic is high then that particular cerely task keeps on running for 10-15 hours.

Answer 1

I've found that the most reliable way to do this is first to set global limits in settings.py, where the value is an int in seconds:

 CELERY_TASK_SOFT_TIME_LIMIT = 3540
 CELERY_TASK_TIME_LIMIT = 3600

Then, you can catch the soft time limit exception and clean up your task before the hard limit hits:

from celery.exceptions import SoftTimeLimitExceeded

@shared_task
def mytask():
    try:
        return do_work()
    except SoftTimeLimitExceeded:
        cleanup_in_a_hurry()

See the celery settings docs for more details.

You then override soft_time_limit and time_limit when you define individual tasks, for example:

@shared_task(soft_time_limit=60, time_limit=70)
def mytask()
    ...

There are some more details in the task docs.

Answer 2

You can set up the task time limits as per documentation

If you are going to set it on particular task you can use task_time_limit or task_soft_time_limit depending on your usage characteristics