I have an Airflow DAG scheduled to run daily. When I start a backfill for the last month, Airflow will start processing the runs from oldest to newest. As a single run takes a couple of hours, which means that when a new run becomes available (a day has passed while working through the backfill), the new run will only be processed after the entire backfill has completed (causing recent data to be not available for the company). Is it possible to instruct Airflow to process runs from most recent to oldest?
You can do it in Airflow 1.10.3
https://airflow.apache.org/cli.html#backfill
airflow backfill --run_backwards dag_id
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With