How to diagnose Celery workers that hang

Question

I am getting started with Celery for a Django project. For local development purposes I've set it up with djcelery and djkombu (database transport), according to these instructiuons http://ask.github.com/celery/tutorials/otherqueues.html#django-database but plan on using rabbitmq on the production server.

I've found that it works but that one or more of the workers will hang at random. The workers hang on different tasks, so it's not a case that one particular task is entering an infinite loop. When I try to shut down celery, in the log I usually see all the workers exiting except for one. Both before and after a worker hangs, I've tried using celeryctl to get information about the workers, but celeryctl always fails with a message "Error: No nodes replied within time constraint. ". I've also tried inspecting workers http://docs.celeryproject.org/en/latest/userguide/workers.html#inspecting-workers but all of the methods return nothing.

Any ideas about how to diagnose what's causing the worker to hang or would be helpful.

Answer 1

I was having the same problem. I think this is because you're using ghettoq which does not support events.

According to ask:

remote control commands (broadcast) and events does not work with ghettoq.