Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Apache Airflow pools: used slots > available slots

Tags:

airflow

I have 6 subdags. Each of them contains a task with pool='crawler' that requires a lot of resources so I created a pool crawler with only 1 slot.

When I run the DAG it seems that the pool restriction is bypassed and all six tasks are executed at the same time (as you can see from the screenshot).

How can I force used slots to be <= available slots?

airflow pools

like image 799
vinsce Avatar asked Nov 07 '18 16:11

vinsce


1 Answers

From the source code:

Airflow pool is not honored by SubDagOperator. Hence resources could be consumed by SubdagOperators

like image 121
kaxil Avatar answered Sep 28 '22 02:09

kaxil