Check status of Celery worker

Question

I have a project that uses Celery. I am periodically running into a scenario where my requests are making it to Celery but the tasks aren't being handed off to the workers, but rather the server is just returning a 500 error.

When I restart Celery it starts working again. I am only guessing that the worker is hanging which makes it so there aren't anymore workers available. If I startup another batch of workers the requests start working again (which supports my theory).

Questions:

1. I understand celery by default logs to stderr. I am not seeing any errors, so I am hoping there is another celery log somewhere. Where would that be?
2. Is there any way to lookup the status of workers? Are they available? Are they hung?
3. Could it be anything else?

Answer 1

you can see active workers using flower API or by directly querying celery using below manner:

from celery import Celery
app = Celery('vwadaptor',
             broker='redis://workerdb:6379/0',
             backend='redis://workerdb:6379/0')

app.control.inspect().active()

Shows you the active workers and their currently executing jobs.

Answer 2

To get a quick status of your workers you can run this command:

celery -A project inspect stats

it will output a JSON array with lot statistics about your workers.

See docs for more details.