How to stop/kill airflow scheduler started in daemon mode

Question

I and new to airflow and accidentally started airflow scheduler in daemon mode. Now, I want to kill the scheduler and possibly restart it. I tried doing

    sudo kill -9 <list of pids>
    pkill <name>

nothing is happening. When I run

    ps aux | grep 'airflow scheduler'

I see these entries:

    user1   2907  6.0  1.0 329788 62996 ?        Sl   17:37   1:26 /users/user1/anaconda2/bin/python /users/user1/anaconda2/bin/airflow scheduler -D
    user1   2909  0.0  0.9 327576 58948 ?        Sl   17:37   0:00 /users/user1/anaconda2/bin/python /users/user1/anaconda2/bin/airflow scheduler -D
    user1   2910  0.0  0.9 327576 58944 ?        Sl   17:37   0:00 /users/user1/anaconda2/bin/python /users/user1/anaconda2/bin/airflow scheduler -D
    user1   2911  0.0  0.9 327576 58944 ?        Sl   17:37   0:00 /users/user1/anaconda2/bin/python /users/user1/anaconda2/bin/airflow scheduler -D

...and so on for 35 lines with different pids.

Any recommendation as to how I can stop/kill airflow scheduler without restarting my machine. I have also checked the pid file for scheduler and tried killing that pid but no effects.

Any help is appreciated. Thanks!

Answer 1

Unfortuntely

kill $(ps -ef | grep "airflow scheduler" | awk '{print $2}')

I was not able to find a clean solution.

Also looking into the code

https://github.com/apache/incubator-airflow/blob/master/airflow/bin/cli.py

Answer 2

Another alternative is:

/usr/bin/rm -f ${AIRFLOW_HOME}/airflow-scheduler.pid
/usr/bin/pkill -f "airflow-scheduler"

Please note better to remove older daemon .pid file; to avoid any issue while restarting on daemon mode.

You can also see: https://github.com/apache/airflow/issues/77

/bin/sh -c 'PATH=/bin:/sbin:/usr/bin:/usr/sbin  mPPID=`cat ${AIRFLOW_HOME}/airflow-scheduler.pid`;ps -o pid= --ppid $mPPID | xargs kill -15 && kill -15 $mPPID && rm -f ${AIRFLOW_HOME}/airflow-scheduler.pid'"

But this command relies on parent PID (.pid file) so if parent process was removed and child processes are still running it won't work.

So in my opinion accepted answer is the best. Or if you have installed use pkill

If using any monitoring service i.e monit like in the github link. Valid stop command would be:

gPPID=`ps -ef | /usr/bin/grep 'airflow scheduler' | /usr/bin/grep -v grep | awk '{print $2}' `; echo $gPPID; | xargs kill -15 && rm -f ${AIRFLOW_HOME}/airflow-scheduler.pid