How to make celery retry using the same worker?

Question

I'm just starting out with celery in a Django project, and am kinda stuck at this particular problem: Basically, I need to distribute a long-running task to different workers. The task is actually broken into several steps, each of which takes considerable time to complete. Therefore, if some step fails, I'd like celery to retry this task using the same worker to reuse the results from the completed steps. I understand that celery uses routing to distribute tasks to certain server, but I can't find anything about this particular problem. I use RabbitMQ as my broker.

Answer 1

You could have every celeryd instance consume from a queue named after the hostname of the worker:

celeryd -l info -n worker1.example.com  -Q celery,worker1.example.com

sets the hostname to worker1.example.com and will consume from a queue named the same, as well as the default queue (named celery).

Then to direct a task to a specific worker you can use:

task.apply_async(args, kwargs, queue="worker1.example.com")

similary to direct a retry:

task.retry(queue="worker1.example.com")

or to direct the retry to the same worker:

task.retry(queue=task.request.hostname)