Stopping Supervisor doesn't stop Celery workers

Question

How do I ensure sub-processes are stopped when I stop Supervisord?

I'm using Supervisord to run two Celery workers. The command for each worker is:

command=/usr/local/myapp/src/manage.py celery worker --concurrency=1 --loglevel=INFO

When I start supervisord, they run perfectly, but when I run sudo service supervisord stop, it reports my workers stop:

celery_1: stopped
celery_2: stopped

However, when I run ps aux | grep celery I see they're still running:

www-data 27667  0.0  1.5 318420 64360 ?        S    10:45   0:00 manage.py celery worker --concurrency=1 --loglevel=INFO
www-data 27668  0.0  1.5 318420 318416 ?        S    10:45   0:00 manage.py celery worker --concurrency=1 --loglevel=INFO

Answer 1

i believe stopping the supervisor service stops the supervisor daemon, not the supervisor process that is managing your celeryworkers

supervisorctl stop all should allow you stop the workers, and also allow you to start/restart them

http://supervisord.org/running.html#running-supervisorctl

Answer 2

The reason I ended up here is that when I was using supervisor to start and stop my celery processes, the celery workers were not stopping, which lead to an accumulation of workers.

I tried various settings in supervisor, such as stopsignal, and killasgroup. None of them stopped the workers.

At this point, it appears that the celery process does not send signals to the workers when it is stopped.

What I ended up doing was to add:

ps auxww | grep 'celery worker' | awk '{print $2}' | xargs kill -9

to my scripts. This is from the celery docs

At the end of this command, I tried kill -TERM instead of kill -9, hoping that would kill the processes more gracefully. But that always left one process.