When to upgrade Airflow's executor from LocalExecutor to CeleryExecutor?

Question

I am currently running several Airflow DAGs using LocalExecutor and it's been working fine. My server has plenty of resources. I am about to add a new DAG for a larger project and I am considering switching from LocalExecutor to CeleryExecutor.

My question is, what are some signs that I should switch to CeleryExecutor? Are there specific performance metrics I should be looking at to know when I need to start scaling out?

Answer 1

The LocalExecutor should be sufficient if you want to run a small number of concurrent tasks such that with your scheduling needs those tasks could run a single server. You should think of moving to CeleryExecutor when you feel that you need to distribute the jobs across multiple servers.