apache airflow define data for all scripts in DAG folder

Question

I have data like Amazon S3 connection settings etc.. which is relevant for many of the .py files in my DAG folder.

I'm aware that default_args is shared with all DAGS in a specific .py file My question is how can I define data which will be accessible for ALL .py files on my DAG folder?

Imagine that I need to change my S3 connection details... I don't want to go over all files and change it one by one. I want to change it just in one place.

Does Airflow support this?

Answer 1

This is precisely the use case for the Connection model. Instead of passing around sensitive information, namely credentials, you create a Connection record which holds that information on your behalf and which you reference with a string.

You can read more about them here.