Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to instruct airflow to backfill from most recent to oldest

I have an Airflow DAG scheduled to run daily. When I start a backfill for the last month, Airflow will start processing the runs from oldest to newest. As a single run takes a couple of hours, which means that when a new run becomes available (a day has passed while working through the backfill), the new run will only be processed after the entire backfill has completed (causing recent data to be not available for the company). Is it possible to instruct Airflow to process runs from most recent to oldest?

like image 704
zeebonk Avatar asked Jul 18 '18 11:07

zeebonk


1 Answers

You can do it in Airflow 1.10.3

https://airflow.apache.org/cli.html#backfill

airflow backfill --run_backwards dag_id
like image 85
shankshera Avatar answered Jan 03 '23 08:01

shankshera