Menu
I have retry logic for tasks and it's not clear how Airflow handles task failures when retries are turned on.
Their documentation just states that on_failure_callback gets triggered when a task fails, but if that task fails and is also marked for retry does that mean that both the on_failure_callback and on_retry_callback would be called?
296
If you want to re-run a task in Airflow, the best way to do so is to press Clear or Delete (language depends on the Airflow version you're running), not Run . Hitting this will clear the state of your failed task and allow the scheduler to pick it back up and re-run it.
Retry logic/parameters will take place before failure logic/parameters. So if you have a task set to retry twice, it will attempt to run again two times (and thus executing on_retry_callback ) before failing (and then executing on_failure_callback ).
If you want to control your task's state from within custom Task/Operator code, Airflow provides two special exceptions you can raise: AirflowSkipException will mark the current task as skipped. AirflowFailException will mark the current task as failed ignoring any remaining retry attempts.
Retry logic/parameters will take place before failure logic/parameters. So if you have a task set to retry twice, it will attempt to run again two times (and thus executing on_retry_callback ) before failing (and then executing on_failure_callback).
An easy way to confirm the sequence that it is executed in is to set your email_on_retry and email_on_failure to True and see the order in which they appear. You can physically confirm that it will retry before failing.
default_args = {
'owner': 'me',
'start_date': datetime(2019, 2, 8),
'email': ['[email protected]'],
'email_on_failure': True,
'email_on_retry': True,
'retries': 1,
'retry_delay': timedelta(minutes=1)
}
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
