Airflow Passing DataFrames Between Python Operators

Question

I tried looking for this answer but could not find anything that makes sense for what I am trying to do.

I am trying to create a DAG that will grab a file from a FTP and return a data frame, clean the data frame, and then import to SQL. So it will end up looking like:

connect_to_ftp >> get_file >> clean_dataframe >> import_to_sql

So if I wanted to get the dataframe in one operator, and the pass it into another, how would I do this? Do I use xcom variables, or should I just make global variables in my code?

Answer 1

So I read in the docs: https://airflow.apache.org/concepts.html#operators

Operators should be atomic and should rarely pass data to each other. So I guess im thinking about it wrong?