Apache Airflow ignore failed task

Question

Is there a way to ignore failed task and proceed to next step after let's say 2 re-tries?

Example;

t1= SomeOperator(...)
t2= SomeOperator(...)

t2.set_upstream(t1)


# if t1 fails re-try 2 times and proceed to t2

# else if t1 success then proceed to t2 as usual 

Answer 1

Take a look at airflows trigger rules.

By default, the trigger rule for every task is 'all_success', meaning the task will only get executed when all directly upstream tasks have succeeded.

What you would want here is the trigger rule 'all_done', meaning all directly upstream tasks are finished, no matter whether they failed or succeeded.

But be careful, as this also means that if a task that is not directly upstream fails, and the tasks following that task get marked as 'upstream_failed', the task with this trigger rule will still get executed.

So in your case, you would have to set retries=2 for t1 and trigger_rule='all_done' for t2.