reliable way to deploy new code into a production celery cluster without pausing service

Question

I have a few celery nodes running in production with rabbitmq and I have been handling deploys with service interruption. I have to take down the whole site in order to deploy new code out to celery. I have max tasks per child set to 1, so in theory, if I make changes to an existing task, they should take effect when the next time they are run, but what about registering new tasks? I know that restarting the daemon won't kill running workers, but instead will let them die on their own, but it still seems dangerous. Is there an elegant solution to this problem?

Answer 1

The challenging part here seems to be identifying which celery tasks are new versus old. I would suggest creating another vhost in rabbitmq and performing the following steps:

1. Update django web servers with new code and reconfigure to point to the new vhost.
2. While tasks are queuing up in the new vhost, wait for celery works to finish up with the tasks from the old vhost.
3. When workers have completed, update the code and configuration to the new vhost

I haven't actually tried this but I don't see why this wouldn't work. One annoying aspect is having to alternate between the vhosts with each deploy.