Composer auto scaling?

Question

Given that GCP Cloud Composer is running with GKE/GCE, is it auto scaling?

Now I have 3 nodes in the cluster to support say, 100 DAGs.

Later, if I have about 300 DAGs, will it scale up itself (w/ celery workers)?

Answer 1

We currently don't support auto-scaling but it's on our roadmap. You could however manually scale up/down the GKE cluster by updating the nodeCount value.

Answer 2

I just published an article showing how to enable autoscale in its underlying kubernetes https://link.medium.com/AMUlwUIkD0.

Basically:

1. Enable autoscale on the node level

2. Apply HorizontalPodAutoscaler to airflow-worker deployment

3. Increase some airflow config parameter to remove the bottleneck

This has been tested in composer-1.7.2 and composer-1.7.5 but might be applicable to other version as well

Do check it out