I have 6 subdags. Each of them contains a task with pool='crawler'
that requires a lot of resources so I created a pool crawler
with only 1 slot.
When I run the DAG it seems that the pool restriction is bypassed and all six tasks are executed at the same time (as you can see from the screenshot).
How can I force used slots to be <=
available slots?
From the source code:
Airflow pool is not honored by SubDagOperator. Hence resources could be consumed by SubdagOperators
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With