Scheduling dag runs in Airflow

Question

Got a general query on Airflow

Is it possible to have a dag file scheduled based on another dag file's schedule.

For example, if I have 2 dags namely dag1 and dag2. I am trying to see if I can have dag2 run each time dag1 is successful else dag2 does not run. Is this possible in Airflow.

Answer 1

You will want to add a TriggerDagRunOperator the end of dag1 and set the schedule of dag2 to None.

In addition, if you want to handle multiple cases for the output of dag1, you can add in a BranchPythonOperator to create multiple paths based on its output. For example, you could set it to either execute the TriggerDagRunOperator on success or Slack you "Warning! Task Failure in DAG1!" with the SlackAPIPostOperator if an error is thrown (or any other logic you want to build in).

If you don't care about multiple outcomes, you could also just use the ShortCircuitOperator before the TriggerDagRunOperator to prevent it from running based on the dag1 output.

Answer 2

Yes. You should include a task at the end of dag1 using the TriggerDagRunOperator. Make sure this task has a trigger rule that will only allow it to run if all other tasks upstream succeed.

The other answer recommends subdags which can have some oddities that make them less than ideal in complex Airflow environments.