Monitoring for Huey (Redis Queue, Time in Queue, and # of Workers)

Question

Switched out Celery for Huey and am loving everything about it :)

One thing Celery has that I do miss is its flower monitoring tool. Are there any monitoring tools that exist for Huey?

We need to track # of tasks in the Redis Queue, Total time a task spends enqueued before being picked up by a worker, and how many workers are running.

Are there any Huey hooks that could help with this?

I've looked through Huey events but they seem to all only fire once the consumer has picked up a task.

My plan right now is:

1. Pass a timestamp to the task as an argument when it is enqueued, and then I can compare it to the timestamp once it is pulled out of the queue by a worker. This would get Time in Queue.

2. I can make a service that calls LLEN on the Redis Huey Queue to track jobs.

3. I'm not sure the best way to get # of workers. Ideally we want to scale up the number of instances with Huey workers if our task queue gets to a certain length.

Does anyone have any experience monitoring Huey? Is there something I'm missing in the documentation that could help this process?

Answer 1

Huey has signals which can help with monitoring certain lifecycle events that occur with each task - e.g. scheduled, triggered, completed, errored. If you timestamp these events, you can determine the time spent in each stage. The SIGNAL_ERROR hook is particularly useful for alerting via Sentry or Slack.

I have not dug into monitoring the number of threads/processes currently being used but I am sure it's doable. Pretty simple monitor dashboard I built below.

Answer 2

I have managed to find a solution to monitor working processes. I needed to count the working processes at a time. So, I did it through the mechanism of signals.

Huey provides signal decorators to fire on a certain signal, e.g. SIGNAL_EXECUTING, SIGNAL_COMPLETE, SIGNAL_ERROR.

On every executing process, a new key with certain prefix is stored in Redis hash. When the process ends the key is deleted. So, to get the number of executing tasks, you can call the get_executing_count().

My working code:

from huey import RedisHuey
from huey.constants import EmptyData
from huey.signals import SIGNAL_EXECUTING, SIGNAL_COMPLETE, SIGNAL_ERROR, SIGNAL_REVOKED

EXECUTING_PREFIX = "executing"

redis_addr = os.getenv("REDIS", "localhost")
huey = RedisHuey('entrypoint', host=redis_addr)

def get_executing_count():
    global huey
    matching = list(
        filter(
            lambda key: key.decode().startswith(f"{EXECUTING_PREFIX}-"), 
            huey.storage.conn.hgetall(huey.storage.result_key).keys()
        )
    )
    return len(matching)

@huey.signal(SIGNAL_EXECUTING)
def task_signal_executing(signal, task):
    global huey
    huey.storage.put_data(f"{EXECUTING_PREFIX}-{task.id}", 1)

@huey.signal(SIGNAL_COMPLETE)
def task_signal_complete(signal, task):
    global huey
    huey.storage.delete_data(f"{EXECUTING_PREFIX}-{task.id}")

@huey.signal(SIGNAL_ERROR, SIGNAL_REVOKED)
def task_signal_error(signal, task, exc=None):
    global huey
    huey.storage.delete_data(f"{EXECUTING_PREFIX}-{task.id}")

Value retrieval I do the following way

from entrypoint import get_executing_count

get_executing_count()