Why is there a DAG named 'airflow_monitoring' automatically generated in Cloud Composer?

Question

When creating an Airflow environment on GCP Composer, there is a DAG named airflow_monitoring automatically created and that comes back even when deleted.

Why? How to handle it? Should I copy this file inside my DAG folder and resign myself to make it part of my code? I noticed that each time I upload my code it stops the execution of this DAG as it could not be found inside the DAG folder until it magically reappears.

I have already tried deleting it inside the DAG folder, delete the logs, delete it from the UI, all of this at the same time etc.

Answer 1

The airflow_monitoring DAG is a per-environment liveness prober/healthcheck that is used to populate the Cloud Composer monitoring metric environment/healthy. It is an indicator for the general overall health of your environment, or more specifically, its ability to schedule DAGs and run tasks. This allows you to use Google Cloud Monitoring features such as metric graphs, or setting alerts when your environment becomes unhealthy.

You can find more information about the metric on the GCP Metrics List, and can explore the metric in Cloud Monitoring under the following:

• Resource type: Cloud Composer Environment
• Metric: Healthy

This is a Composer-managed DAG and uses very minimal resources from your environment. Ideally, you should leave it untouched, as it has little to no effect on anything else running in your environment.