Menu
Can anyone help me understand the 'schedule' column in the airflow web UI.
Why does it sometimes colour the icon red? Is that showing the scheduler is down? It seems like this is the case, but I couldn't find any explanation definitive explanation anywhere about this column.
883
The Airflow scheduler monitors all tasks and DAGs, then triggers the task instances once their dependencies are complete. Behind the scenes, the scheduler spins up a subprocess, which monitors and stays in sync with all DAGs in the specified DAG directory.
To schedule a dag, Airflow just looks for the last execution date and sum the schedule interval . If this time has expired it will run the dag. You cannot simple update the start date. A simple way to do this is edit your start date and schedule interval , rename your dag (e.g. xxxx_v2.py) and redeploy it.
Airflow loads DAGs from Python source files, which it looks for inside its configured DAG_FOLDER . It will take each file, execute it, and then load any DAG objects from that file. This means you can define multiple DAGs per Python file, or even spread one very complex DAG across multiple Python files using imports.
It's related to that DAG reaching the DAG run limit. The DAG run limit defaults to core:max_active_runs_per_dag, but can be overridden by the max_active_runs parameter when creating a DAG.
If you hover over the red label, it should actually show you the number of currently active runs and the limit. Clicking it should also take you to the list of DAG runs for that DAG, which is a useful view to override the state if necessary (i.e. set every run as failed).
Given that your schedule is None, I assume you may have kicked off too many manual runs of the DAG at once or need to bump up the limit.
146
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
