HEALTHCHECK of a Docker container running Celery tasks?

Question

I know one of the ways to check health for Docker container is using the commmand

HEALTHCHECK CMD curl --fail http://localhost:3000/ || exit 1

But in case of workers there is no such URL to hit , How to check the container's health in that case ?

Answer 1

The celery inspect ping command comes in handy, as it does a whole trip: it sends a "ping" task on the broker, workers respond and celery fetches the responses.

Assuming your app is named tasks.add, you may ping all your workers:

/app $ celery inspect ping -A tasks.add
-> celery@aa7c21dd0e96: OK
        pong
-> celery@57615db15d80: OK
        pong

With aa7c21dd0e96 being the Docker hostname, and thus available in $HOSTNAME.

To ping a single node, you would have to run:

celery inspect ping -A tasks.add -d celery@$HOSTNAME

Here, d stands for destination.

The line to add to your Dockerfile:

HEALTHCHECK CMD celery inspect ping -A tasks.add -d celery@$HOSTNAME

---

Sample outputs:

/app $ celery inspect ping -A tasks.add -d fake_node
Error: No nodes replied within time constraint.
/app $ echo $?
69

Unhealthy if the node does not exist or does not reply

/app $ celery inspect ping -A tasks.add -d celery@$HOSTNAME
-> celery@d39b3d31cc13: OK
        pong
/app $ echo $?
0

Healthy when the node replies pong.

/app $ celery inspect ping -d celery@$HOSTNAME
Traceback (most recent call last):
...
    raise socket.error(last_err)
OSError: [Errno 111] Connection refused
/app $ echo $?
1

Unhealthy when the broker is not available - I removed the app, so it tries to connect to a local AMPQ and fails This might not suit your needs, the broker is unhealthy, not the worker.

Answer 2

The below example snippet, derived from that posted by @PunKeel, is applicable for those looking to implement health check in docker-compose.yml which could be used through docker-compose or docker stack deploy.

worker:
    build:
        context: .
        dockerfile: Dockerfile
    image: myimage
    links:
        - rabbitmq
    restart: always
    command: celery worker --hostname=%h --broker=amqp://rabbitmq:5672
    healthcheck:
        test: celery -b amqp://rabbitmq:5672 inspect ping -d celery@$$HOSTNAME
        interval: 30s
        timeout: 10s
        retries: 3

Notice the extra $ in the command, so that $HOSTNAME actually gets passed into the container. I also didn't use the -A flag.

Ideally, rabbitmq should also have its own health check, perhaps with curl guest:guest@localhost:15672/api/overview, since docker wouldn't be able to discern if worker is down or the broker is down with celery inspect ping.