Celery's expires option doesn't work

Question

I'm playing around with Celery, and I'm trying to do a periodic task with CELERYBEAT_SCHEDULER. Here is my configuration:

CELERY_TIMEZONE = 'Europe/Kiev'
CELERYBEAT_SCHEDULE = {
    'run-task-every-5-seconds': {
        'task': 'tasks.run_every_five_seconds',
        'schedule': timedelta(seconds=5),
        'options': {
            'expires': 10,
        }
    },
}

# the task
@app.task()
def run_every_five_seconds():
   return '5 seconds passed'

When running the beat with celery -A celery_app beat the task doesn't seem to expire. Then I've read that there might be some issue with the beat, so it do not take into account the expires option.

Then I've tried to do a task, so it gets called manually.

@app.task()
def print_hello():
    while True:
        print datetime.datetime.now()
        sleep(1)

I am calling the task in this way:

print_hello.apply_async(args=[], expires=5)

The worker's console is telling my that my task will expire, but it doesn't get expired as well. It's getting executed infinitely.

Received task: tasks.print_hello[05ee0175-cf3a-492b-9601-1450eaaf8ef7] expires:[2016-01-15 00:08:03.707062+02:00]

Is there something I am doing wrong?

Answer 1

I think you have understood the expires argument wrong.

The documentation says: "The task will not be executed after the expiration time." ref. It means the execution will not start if the expiration time has passed. If the execution has already started, the execution will run to completion.

Your configuration adds a task to task queue every 5 seconds. If the execution does not start in 10 seconds from the time the task is added to the task queue, the task is discarded. However, the tasks is executed immediately because there is a free celery worker available.

Your code example adds a task that is discarded if the execution is not started in 5 seconds.

To get the functionality you want, you can replace 'expires': 10, with 'expires': datetime.datetime.now() + timedelta(seconds=10),. That will set the expires to a absolute time.