Restarting the airflow scheduler

Question

I'm trying to get airflow working to better orchestrate an etl process. When I make changes to a dag in my dags folder, I often have to restart the scheduler with

airflow scheduler

before the changes are visible in the UI. I would like to run the scheduler as a daemon process with

airflow scheduler -D

but we I try to do so, I get a message saying

[2018-10-17 14:13:54,769] {jobs.py:580} ERROR - 
Cannot use more than 1 thread when using sqlite. Setting max_threads to 1

I think this error pops up because the scheduler is already running as a daemon. However, when I try to find out where the scheduler is being run with

lsof -i

I don't get any results.

Question: Why am I not able to restart the scheduler with airflow scheduler -D. Why does the scheduler restart with airflow webserver? How do I successfully kill the process that is preventing me to run airflow scheduler -D?

Answer 1

Run ps aux | grep airflow and check if airflow webserver or airflow scheduler processes are running. If they are kill them and rerun using airflow scheduler -D

Answer 2

You need to clear out the airflow-scheduler.pid file at $AIRFLOW_HOME. The stale pid file from the daemon will prevent you to start another scheduler process.