Menu
I'm trying to figure out a way to test a DAG where I have a couple of tasks communicating using XCom.
Since the console command only allow me to run tasks from a DAG, is there a way to test the communication without having to run the DAG via the UI?
Thanks
650
What is an Airflow XCom ? XCom stands for “cross-communication” and allows to exchange messages or small amount of data between tasks. You can think of an XCom as a little object with the following fields: that is stored IN the metadata database of Airflow.
Storing Variables in Environment Variables Airflow Variables can also be created and managed using Environment Variables. The environment variable naming convention is AIRFLOW_VAR_{VARIABLE_NAME} , all uppercase. So if your variable key is FOO then the variable name should be AIRFLOW_VAR_FOO .
Here's a way that worked for me.
Even though the Airflow web page states that the test command does not generate or keeps any state, running the airflow test command in sequence worked.
Basically you do:
Airflow test my_dag task1 date
Airflow test my_dag task2 date
where task1 returns the value or uses xcom_push method to send the value and then task2 run xcom_pull to get the value and move on.
It worked for me. Please comment if you have different approaches or other questions.
197
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With