Monitor a celery task state without polling?

Question

Is it possible to monitor the state of a celery task without polling?

For example, if I have a task which periodically updates its state using update_state:

@task(bind=True)
def my_task(self):
    for x in range(100):
        time.sleep(1)
        self.update_state(state='PROGRESS', meta={'x': x})

Is it possible to monitor that state from another process without polling?

Answer 1

I haven't done this myself yet, so this isn't a complete answer, but I have interest in solving the same problem. I've got three ideas worth consideration:

• Would using the built-in handler states like on_failure, on_retry, and on_success work for your use case?

• Another is to subclass Task (trivial example here: Callback for celery apply_async) and add on_* handlers that get passed callback functions for your custom states. Perhaps that wouldn't even be necessary

• The best might be to write a custom event receiver. I think this is how their "Real-time Celery web-monitor" example works. A related SO post on this is: Implementing Twisted style local multiple deferred callbacks in Celery.