Whenever I run "stop celery-worker" in Supervisor to stop my Celery workers I get the following errors:
625[2014-05-18 14:41:59,650: ERROR/MainProcess] Task tasks.send_process_my_task[2ff51473-80f9-4c53-9b9a-b8ddc8c36618] raised unexpected: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
626Traceback (most recent call last):
627 File "/usr/local/envs/mylocalenv/local/lib/python2.7/site-packages/billiard/pool.py", line 1168, in mark_as_worker_lost
628 human_status(exitcode)),
629WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).
630[2014-05-18 14:41:59,653: ERROR/MainProcess] Process 'Worker-4' pid:2279 exited with exitcode -15
631[2014-05-18 14:41:59,683: ERROR/MainProcess] Task tasks.send_process_my_task[9847bda0-9073-4b66-872f-ab4ed18b8690] raised unexpected: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
However, if I stop my worker in the foreground using Ctrl-C, the workers shut down gracefully without any error:
[2014-05-18 14:53:39,124: DEBUG/MainProcess] | Worker: Stopping Pool...
190[2014-05-18 14:53:46,769: INFO/MainProcess] Task tasks.send_process_my_task[8071162a-b47a-4aa4-94e7-72f51443b411] succeeded in 16.467172019s: u''
191[2014-05-18 14:53:46,816: INFO/MainProcess] Task tasks.send_process_my_task[1dd4cb3c-322c-41a3-94ea-f0dde604e6d3] succeeded in 16.389319943s: u''
192[2014-05-18 14:53:46,817: INFO/MainProcess] Task tasks.send_process_my_task[5e72e357-352b-47aa-9dda-51da53220a5e] succeeded in 16.424921152s: u''
193[2014-05-18 14:53:48,903: INFO/MainProcess] Task tasks.send_process_my_task[30c1064a-0f04-49e2-9282-a6031e97a448] succeeded in 18.595272409s: u''
Is there anything in supervisor that would lead to killing the workers that way? I expected supervisor to send a sigterm signal and to not kill the processes.
Here is a copy of my celery supervisor configuration:
[program:celery-worker]
command=/path/to/script-running-celery.sh
directory=/path/to/celery/project/tasks
user=myuser
stdout_logfile=/var/log/celery/worker.log
stderr_logfile=/var/log/celery/worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 1200
killasgroup=true
stopasgroup=true
priority=50
The script executed from supervisor (which runs celery):
#!/bin/bash
export PATH="$PATH:/mylocal-envs/localenv";
cd /path/to/celery/project/tasks;
exec /mylocal-envs/localenv/bin/celery worker -c8 -n my_worker_%h_$(uuid) --without-gossip -A tasks --queues=myqueue --loglevel=INFO -Ofair
I figured out that if changing the stopsignal option to INT instead of the default TERM, the worker gets shut down gracefully, as a regular warm shutdown from the foreground when running celery in the terminal.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With