Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is it possible to use a single task in multiple DAGS in airflow?

Tags:

python

airflow

I have a task_a that I want to use in DAG_1 and DAG_2. Is this possible in airflow?

task_a = SomeOperator(
task_id='some_id',
bash_command='some_command',
#instead of just
dag= DAG_1 # I want to assign this task to multiple dags
#dag=assign_multiple_dags_here(DAG_1 and DAG_2)
)

Is this possible?

like image 438
javed Avatar asked Feb 04 '23 18:02

javed


2 Answers

you could always do something with partial and then assign it to 2 different dags:

from functools import partial
task_template = partial(SomeOperator, some_id='id', some_command='cmd')
task_template(dag=dag1)
task_template(dag=dag2)

you could also just create a function that does it:

def create_task(dag):
    return SomeOperator(some_id='id', some_command='cmd', dag=dag)

for d in (dag1, dag2):
    create_task(d)
like image 89
acushner Avatar answered Feb 07 '23 06:02

acushner


As per Current Design no.

Tasks are part of DAG's. Each DAG run creates a task instance.

This is to keep the framework's housekeeping simple

like image 23
Priyank Mehta Avatar answered Feb 07 '23 08:02

Priyank Mehta