Airflow/Composer recommended folder structure

Question

Do you guys have any recommended for Composer folder/directories structure? The way it should be structured is different from the way our internal Airflow server is using right now.

Based on Google documentation: https://cloud.google.com/composer/docs/concepts/cloud-storage:

• plugins/: Stores your custom plugins, operators, hooks
• dags/: store dags and any data the web server needs to parse a dag.
• data/: Stores the data that tasks produce and use.

This is an example of how I organize my dags folder:

I had trouble before when I put the key.json file in the data/ folder and the dags cannot be parsed using the keys in the data/ folder. So now I tend to put all the support files in the dags/ folder.

Would the performance of the scheduler be impacted if I put the supported files (sql, keys, schema) for the dag in the dags/ folder? Is there a good use case to use the data/ folder?

It would be helpful if you guys can show me an example of how to structure the composer folder to support multiple projects with different dags, plugins and supported files.

Right now, we only have 1 Github for the entire Airflow folder. Is it better to have a separate git per project?

Thanks!

Answer 1

The impact on the scheduler should be fairly minimal as long as the files you place in the dags folder are not .py files; however, you can also place the files in the plugins folder which is also synced via copy.

I would use top-level folders to separate projects (e.g. dags/projectA/dagA.py), or even separate environments if the projects are large enough.