Menu
I am new to airflow and wanted to know the difference between execution_timeout and dagrun_timeout in Airflow. Actually, in my codebase, I am currently using execution_timeout but some of the dags are not respecting timeout.
532
max_active_runs : This is the maximum number of active DAG runs allowed for the DAG in question. Once this limit is hit, the Scheduler will not create new active DAG runs. If this setting is not defined, the value of the environment-level setting max_active_runs_per_dag is assumed.
Upstream task: A task that must reach a specified state before a dependent task can run. Downstream task: A dependent task that cannot run until an upstream task reaches a specified state.
timedelta) – time by which the job is expected to succeed. Note that this represents the timedelta after the period is closed. For example if you set an SLA of 1 hour, the scheduler would send an email soon after 1:00AM on the 2016-01-02 if the 2016-01-01 instance has not succeeded yet.
The `dag_id` is a unique identifier for your DAG, and the `task_id` is a unique name (within the DAG's namespace) for a task. It's nice if these names are meaningful to you and briefly describe your workflow and individual tasks.
From the documentation:
execution_timeout (datetime.timedelta) – max time allowed for the execution of this task instance, if it goes beyond it will raise and fail.
dagrun_timeout (datetime.timedelta) – specify how long a DagRun should be up before timing out / failing, so that new DagRuns can be created
The execution_timeout refers to the execution of the TaskInstance, while the dagrun_timeout is about the entire DAG which can consist of many tasks.
To understand why your tasks are not respecting the timeout, you will need to provide more information, and ideally a minimal example.
65
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
