How can I make celery die after there are no more tasks to be run?

Question

I have a celery task that is memory-intensive and I want it to run in an expensive AWS server that stays off most of the time.

A celery beat will schedule the task once a day, but it can also be scheduled manually through a web application (== django).

Scheduling the task is a matter of sending a message to a rabbit queue.

Then I would like to issue a command using AWS api to turn on my expensive server, and have it start the workers, wait for them to die after there are no more tasks, and shutdown (and save some money because the server is off).

I know how to script AWS to turn on my server. The problem is: how do I tell celery to run tasks until the queue is empty and then die?

Is there a command parameter I can give it, or do I need to come up with some kind of hack for that?

Answer 1

Celery worker is just a dumb horse. Something else must monitor the worker and trigger an event if the worker is waiting for tasks.

Use Flower to monitor your Celery cluster and their API to consume the worker activity.

Flower has a beautiful dashboard with workers/tasks data, the information is there and they have an API.