Airflow error handling task trigger rule for triggering directly linked task failure

Question

I have an airflow task pipeline as in the diagram. task1_error_handler & task2_error_handler are error handling tasks which should be ran only if task directly linked is failed. I have set ONE_FAILED trigger rule for these tasks. But seems error on task1 triggers both error handlers. I only need to trigger task1_error_handler. All the tasks are custom operators & the task ids ending with status are custom sensors. How should I achieve this ?

Answer 1

An error on task1 is causing both error handlers to occur because task2 is downstream of task1, making task1 a parent of task task2.

With your trigger rule being ONE_FAILED for both task1 and task2, this is causing problems because the the definition of ONE_FAILED is:

fires as soon as at least one parent has failed, it does not wait for all parents to be done

So with that said, you only want the task1_error_handler to trigger if task1 fails. This cannot be easily done by just changing the trigger rule unfortunately because you can't directly link a conditional task like you want to currently.

Your best bets would be:

• Keep task1 as it is and to get rid of task2's error handler trigger rule and instead use on_failure_callback to call the error handler.
• Split task2 into a separate DAG.