Faster alternative to apache airflow for workflows with many tasks

Question

I currently use Apache Airflow for running data aggregation and ETL workflows. My workflows are fairly complex with one workflow having 15-20 tasks and have branches. I can combine them but doing so would negate the features like retry, execution timers that I use. Airflow works well except that it is quite slow with so many tasks. It takes lot of time between tasks.

Is there an alternative which can execute the tasks faster without gaps in between tasks? I also would like to minimize the effort needed to switch over if possible.

Answer 1

I would recommend Temporal Workflow. It has more developer friendly programming model and scales to orders of magnitude larger use cases. It also already used for multiple latency sensitive applications at many companies.

Disclaimer: I'm the tech lead of the Temporal project and the Co-founder/CEO of the associated company.