Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Tasks linger in celery amqp when publisher is terminated

I am using Celery with a RabbitMQ server. I have a publisher, which could potentially be terminated by a SIGKILL and since this signal cannot be watched, I cannot revoke the tasks. What would be a common approach to revoke the tasks where the publisher is not alive anymore?

I experimented with an interval on the worker side, but the publisher is obviously not registered as a worker, so I don't know how I can detect a timeout

like image 404
Daniel Stephens Avatar asked Sep 19 '19 18:09

Daniel Stephens


3 Answers

There's nothing built-in to celery to monitor the producer / publisher status -- only the worker / consumer status. There are other alternatives that you can consider, for example by using a redis expiring key that has to be updated periodically by the publisher that can serve as a proxy for whether a publisher is alive. And then in the task checking to see if the flag for a publisher still exists within redis, and if it doesn't the task returns doing nothing.

like image 82
2ps Avatar answered Oct 20 '22 07:10

2ps


I am pretty sure what you want is not possible with Celery, so I suggest you to shift your logic around and redesign everything to be part of a Celery workflow (or several Celery canvases depends on the actual use-case). My experience with Celery is that you can build literally any workflow you can imagine with those Celery primitives and/or custom Celery signatures.

like image 44
DejanLekic Avatar answered Oct 20 '22 07:10

DejanLekic


Another solution, which works in my case, is to add the next task only if the current processed ones are finished. In this case the queue doesn't fill up.

like image 4
Daniel Stephens Avatar answered Oct 20 '22 07:10

Daniel Stephens