Menu
Hi I'm using Airflow and running in background in EC2. But how do you stop/check logs? I couldn't find the detailed explanation on the documentation.
Thanks!
678
I was able to find the Airflow webserver logs under /var/log/upstart/even though the designated log location was set to /var/log/airflow/. I originally thought the webserver process was not producing any logs but turns out it was just in a different location than specified in the config files.
I do capture scheduler output by launching it as follows: nohup airflow scheduler >> ${AIRFLOW_HOME}/logs/scheduler.log 2>&1 & The Airflow webserver will create a PID file in the AIRFLOW_HOMEfolder so you stop it with kill -TERM $(cat ${AIRFLOW_HOME}/airflow-webserver.pidor use pkill -f "gunicorn: master [airflow-webserver]".
Job/task logs are available via the admin UI. I have given up on webserver logs. I have never seen any output logging when passing logging parameters (-l, --stdout, --stderr) to the airflow webservercommand. I do capture scheduler output by launching it as follows: nohup airflow scheduler >> ${AIRFLOW_HOME}/logs/scheduler.log 2>&1 &
– jhnclvr Sep 20 '17 at 19:50 1 @jhnclvr - Yes, I used -D (especially for the webserver). In airflow 1.8.0 this tends to block for a really long time. The redirects still don't work properly. – saarp Sep 21 '17 at 23:24 I agree, the logs are still mostly empty. I just couldn't get them to appear at all without -D – jhnclvr Sep 22 '17 at 13:49
Job/task logs are available via the admin UI. I have given up on webserver logs. I have never seen any output logging when passing logging parameters (-l, --stdout, --stderr) to the airflow webserver command.
I do capture scheduler output by launching it as follows:
nohup airflow scheduler >> ${AIRFLOW_HOME}/logs/scheduler.log 2>&1 &
The Airflow webserver will create a PID file in the AIRFLOW_HOME folder so you stop it with kill -TERM $(cat ${AIRFLOW_HOME}/airflow-webserver.pid or use pkill -f "gunicorn: master [airflow-webserver]".
56
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
