Airflow 2.0 Issues : Too many airflow supervisor tasks

Question

I installed airflow 2.0 using docker swarm and Celery Executor.
After 1 week, celery workers memory is overflowing by airflow task supervisor (screenshot attached)
Anyone faced such issues ? Any suggestions ?

Answer 1

In Airflow 2.0, there are 2 ways of creating child processes.

1. forking of the parent process (Fast)
2. spawning a new python process using python subprocess (Slow)

By default, airflow 2.0 uses (1) method. Forking the parent process is faster. On the other hand, the child process is not killed after the task completion. Number of child processes keep increasing till the memory exhaused.

I switched to subprocess method (2) by setting execute_tasks_new_python_interpreter = True. Here, each python process is killed and everytime new process is created. This might be slow but memory is effectively utilised.