Menu
Is there a way to run airflow DAG in a loop? When trying to create a cycle (connecting the last component to the upstream of the last one) I got "Cycle detected in DAG. Faulty task: ..." Generally, I have a short flow of 3 BashOperator components which i want to run continually (without any input-output pass from the last component to the first). Thanks!
786
schedule_interval is defined as a DAG arguments, and receives preferably a cron expression as a str, or a datetime. timedelta object. When following the provided link for CRON expressions it appears you can specify it as */5 * * * * to run it every 5 minutes.
Because a DAG cannot have self-loops, its adjacency matrix must have a zero diagonal, so adding I preserves the property that all matrix coefficients are 0 or 1.
As Airflow has its scheduler and it adopts the schedule interval syntax from cron, the smallest data and time interval in the Airflow scheduler world is minute. Inside of the scheduler, the only thing that is continuously running is the scheduler itself.
You should be able to use the TriggerDagRunOperator to rerun the DAG after the last task is finished. Just put it after the last operator and make it trigger the same DAG.
95
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
