Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

WorkerLostError when stopping Celery worker from Supervisor

Whenever I run "stop celery-worker" in Supervisor to stop my Celery workers I get the following errors:

625[2014-05-18 14:41:59,650: ERROR/MainProcess] Task tasks.send_process_my_task[2ff51473-80f9-4c53-9b9a-b8ddc8c36618] raised unexpected: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)  
626Traceback (most recent call last):                                                                                                                                                                              
627  File "/usr/local/envs/mylocalenv/local/lib/python2.7/site-packages/billiard/pool.py", line 1168, in mark_as_worker_lost                                                                                        
628    human_status(exitcode)),                                                                                                                                                                                    
629WorkerLostError: Worker exited prematurely: signal 15 (SIGTERM).                                                                                                                   
630[2014-05-18 14:41:59,653: ERROR/MainProcess] Process 'Worker-4' pid:2279 exited with exitcode -15                                                                                             
631[2014-05-18 14:41:59,683: ERROR/MainProcess] Task tasks.send_process_my_task[9847bda0-9073-4b66-872f-ab4ed18b8690] raised unexpected: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',) 

However, if I stop my worker in the foreground using Ctrl-C, the workers shut down gracefully without any error:

[2014-05-18 14:53:39,124: DEBUG/MainProcess] | Worker: Stopping Pool...
190[2014-05-18 14:53:46,769: INFO/MainProcess] Task tasks.send_process_my_task[8071162a-b47a-4aa4-94e7-72f51443b411] succeeded in 16.467172019s: u''
191[2014-05-18 14:53:46,816: INFO/MainProcess] Task tasks.send_process_my_task[1dd4cb3c-322c-41a3-94ea-f0dde604e6d3] succeeded in 16.389319943s: u''
192[2014-05-18 14:53:46,817: INFO/MainProcess] Task tasks.send_process_my_task[5e72e357-352b-47aa-9dda-51da53220a5e] succeeded in 16.424921152s: u''
193[2014-05-18 14:53:48,903: INFO/MainProcess] Task tasks.send_process_my_task[30c1064a-0f04-49e2-9282-a6031e97a448] succeeded in 18.595272409s: u''

Is there anything in supervisor that would lead to killing the workers that way? I expected supervisor to send a sigterm signal and to not kill the processes.

Here is a copy of my celery supervisor configuration:

[program:celery-worker]
command=/path/to/script-running-celery.sh
directory=/path/to/celery/project/tasks
user=myuser
stdout_logfile=/var/log/celery/worker.log
stderr_logfile=/var/log/celery/worker.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 1200
killasgroup=true
stopasgroup=true
priority=50

The script executed from supervisor (which runs celery):

#!/bin/bash

export PATH="$PATH:/mylocal-envs/localenv";
cd /path/to/celery/project/tasks;
exec /mylocal-envs/localenv/bin/celery worker -c8 -n my_worker_%h_$(uuid) --without-gossip -A tasks --queues=myqueue --loglevel=INFO -Ofair
like image 467
Loic Duros Avatar asked Dec 31 '25 23:12

Loic Duros


1 Answers

I figured out that if changing the stopsignal option to INT instead of the default TERM, the worker gets shut down gracefully, as a regular warm shutdown from the foreground when running celery in the terminal.

like image 157
Loic Duros Avatar answered Jan 03 '26 22:01

Loic Duros



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!