Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Airflow: how to use xcom_push and xcom_pull in non-PythonOperator

Tags:

airflow

I see a lot of examples on how to use xcom_push and xcom_pull with PythonOperators in Airflow.

I need to do xcom_pull from a non-PythonOperator class and couldn't find how to do it.

Any pointer or example will be appreciated!

like image 878
kee Avatar asked Jul 29 '18 14:07

kee


People also ask

How do I use the airflow pull in XCom?

XComs are explicitly “pushed” and “pulled” to/from their storage using the xcom_push and xcom_pull methods on Task Instances. Many operators will auto-push their results into an XCom key called return_value if the do_xcom_push argument is set to True (as it is by default), and @task functions do this as well.

How do you pull data from XCom?

Pulling a XCom with xcom_pull In order to pull a XCom from a task, you have to use the xcom_pull method. Like xcom_push, this method is available through a task instance object. xcom_pull expects 2 arguments: task_ids, only XComs from tasks matching ids will be pulled.

What is TI in airflow?

Specify the ti argument - It stands for task instance, and allows you to pull values stored in Airflow XComs. The xcom_pull() method - It's used to pull a list of return values from one or multiple Airflow tasks.


1 Answers

From inside an operator's execute method:

Push:

self.xcom_push(context, key, value)

Pull:

self.xcom_pull(context, key=key)

If you have a task instance:

Push:

context["ti"].xcom_push(key, value)

Pull:

context["ti"].xcom_pull(key=key)
like image 180
Beau Barker Avatar answered Sep 24 '22 13:09

Beau Barker