Airflow sql alchemy pool size ignored

Question

I am running airlfow local executor with a postgres database and I am getting a: (psycopg2.OperationalError) FATAL: remaining connection slots are reserved

My configs:

sql_alchemy_pool_size = 5 sql_alchemy_pool_recycle = 1800 sql_alchemy_reconnect_timeout = 300

Still I can see that airflow always keeps more connections open than the pool size.

How can we limit Airflow to actually use the pool limit?

airflow_version = 1.10.2; postgres_version = 9.6

Answer 1

You have forks of the main process as workers, each of which manage their own thread pool.

Check the implementation of LocalExecutor; as it is using multiprocessing under the hood. SqlAlchemy will close any open connections upon forking the LocalWorker; but the pool size will be equivalent to the parent, so at max, theoretically you'd have k * (n + 1) connections, where n is your parallelism constant and k is your sql_alchemy_pool_size.