Menu
I am new to Airflow. I am trying to run airflow scheduler as a daemon process, but the process does not live for long. I have configured "LocalExecutor" in airflow.cfg file and ran the following command to start the scheduler.(I am using Google compute engine and accessing server via PuTTY)
airflow scheduler --daemon --num_runs=5 --log-file=/root/airflow/logs/scheduler.log
When I run this command, the airflow scheduler starts and I can see the airflow-scheduler.pid file in my airflow home folder, but the process does not live for long. When I close the PuTTY session and reconnect to the server, I cannot find the scheduler process. Am I missing something? How can I run the airflow scheduler as a daemon process?
306
We first need to download the service definition files from the Apache Airflow GitHub repo, then put them into the correct system directories. We also need to create some folders because the daemon will need them to run correctly.
The PID file for the webserver will be stored in $AIRFLOW_HOME/airflow-webserver. pid or in /run/airflow/webserver.
To start the airflow job scheduler you need to execute the Airflow Scheduler command. It will use the configuration specified in airflow. cfg. The Airflow Jobs Scheduler runs jobs with schedule_interval AFTER the start date, at the END of the period.
I had a similar problem. My airflow scheduler did not keep running as a deamon process when I executed scheduler as deamon:
airflow scheduler -D
But the scheduler did work when I ran it normally. After I deleted the airflow-scheduler.err file and rerun the scheduler as a deamon process it started working:
rm airflow-scheduler.err
airflow scheduler -D
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
