Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to skip task in Airflow operator?

Is there a way for Airflow to skip current task from the PythonOperator? For example:

def execute():
    if condition:
        skip_current_task()

task = PythonOperator(task_id='task', python_callable=execute, dag=some_dag)

And also marking the task as "Skipped" in Airflow UI?

like image 666
zeyger Avatar asked Oct 16 '19 13:10

zeyger


People also ask

How to conditionally skip tasks in an airflow Dag?

How to conditionally skip tasks in an Airflow DAG 1 Use XCom to store the backfill date. I am not using the backfill feature available in Airflow because I need to pass more parameters than just an execution date. 2 Stopping DAG execution conditionally. ... 3 Skip DAG instead of failing the run. ... 4 PythonSensor waits for too long. ...

What are tasks in airflow?

Tasks A Task is the basic unit of execution in Airflow. Tasks are arranged into DAGs, and then have upstream and downstream dependencies set between them into order to express the order they should run in. There are three basic kinds of Task:

How does airflow detect task/process mismatch?

Airflow detects two kinds of task/process mismatch: Zombie tasks are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.

How to run a task from another task in Apache Airflow?

Yes, you just click on task 3. Toggle the check boxes to the right of the run button to ignore dependencies, then click run. From the way Apache Airflow is built, you can write the logic/branches to determine which tasks to run. You cannot start task execution from any task in between.


1 Answers

Figured it out! Skipping task is as easy as:

def execute():
    if condition:
        raise AirflowSkipException

task = PythonOperator(task_id='task', python_callable=execute, dag=some_dag)
like image 189
zeyger Avatar answered Sep 28 '22 23:09

zeyger