celery worker does not retry task after calling retry()

Question

I have a task:

    @celery.task(name='request_task',default_retry_delay=2,acks_late=True)
    def request_task(data):
        try:
            if some_condition:
                request_task.retry()
        except Exception as e:
            request_task.retry()

I use celery with mongodb broker and mongodb results backend enabled.

When task's retry() method is called, neither from conditional statement nor after catching exception, the task is not retried.

in the worker's terminal I get message like this:

[2012-08-10 19:21:54,909: INFO/MainProcess] Task request_task[badb3131-8964-41b5-90a7-245a8131e68d] retry: Task can be retried

What can be wrong?

UPDATE: Finally, I did not solve this question and had to use while loop inside the task, so my tasks never get retried.

Answer 1

I know this answer is too late, but the log message you're seeing means that you're calling the task directly request_task() and not queuing it so it's not running on a worker, so doing this will raise the exception if there is any, or a Retry exception, this is the code from the Task.retry method if you want to look at:

# Not in worker or emulated by (apply/always_eager),
# so just raise the original exception.
if request.called_directly:
    # raises orig stack if PyErr_Occurred,
    # and augments with exc' if that argument is defined.
    raise_with_context(exc or Retry('Task can be retried', None))

Using task.retry() does not retry the task on the same worker, it sends a new message using task.apply_async() so it might be retried using another worker, this is what you should take into account when handling retries, you can access the retries count using task.request.retries and you can also set max_retries option on the task decorator.

By the way using bind=True on the task decorator makes the task instance available as the first argument:

@app.task(bind=True, name='request_task', max_retries=3)
def request_task(self, data):
    # Task retries count
    self.request.retries

    try:
        # Your code here
    except SomeException as exc:
        # Retry the task
        # Passing the exc argument will make the task fail
        # with this exception instead of MaxRetriesExceededError on max retries
        self.retry(exc=exc)

Answer 2

You should read the section on retrying in the Celery docs. http://celery.readthedocs.org/en/latest/userguide/tasks.html#retrying

It looks like in order to retry, you must raise a retry exception.

raise request_task.retry()

That seems to make the retry be handled by the function that decorated your task.