Interact with celery ongoing task

Question

We have a distributed architecture based on rabbitMQ and Celery. We can launch in parallel multiple tasks without any issue. The scalability is good.

Now we need to control the task remotely: PAUSE, RESUME, CANCEL. The only solution we found is to make in the Celery task a RPC call to another task that replies the command after a DB request. The Celery task and RPC task are not on the same machine and only the RPC task has access to the DB.

Do you have any advice how to improve it and easily communicate with an ongoing task? Thank you

EDIT: In fact we would like to do something like in the picture below. It's easy to do the Blue configuration or the Orange, but we don't know how to do both simultaneously. Workers are subscribing to a common Jobs queue and each worker has its own Admin queue declared on an exchange.

EDIT: IF this is not possible with Celery, I'am open to a solution with other frameworks like python-rq.

Answer 1

It look like the Control Bus pattern.

For a better scalability and in order to reduce the RPC call, I recommend to reverse the logic. The PAUSE, RESUME, CANCEL command are push to the Celery tasks through a control bus when the state change occurs. The Celery app will store the current state of the Celery app in a store (could be in memory, on the filesystem..). If task states must be kept even after a stop/start of the app, It will involve more work in order to keep both app synchronized (eg. synchronization at startup).