Is it possible to get the actual start time of a dag in Airflow? By start time I mean the exact time the first task of a dag starts running.
I know I can use macros to get the execution date. If the job is ran using trigger_dag this is what I would call a start time but if the job is ran on a daily schedule then {{ execution_date }}
returns yesterdays date.
I have also tried to place datetime.now().isoformat()
in the body of the dag code and then pass it to a task but this seems to return the time the task is first called rather than when the dag itself started.
Your DAG will be instantiated for each schedule along with a corresponding DAG Run entry in the database backend. If you run a DAG on a schedule_interval of one day, the run stamped 2020-01-01 will be triggered soon after 2020-01-01T23:59. In other words, the job instance is started once the period it covers has ended.
The start date is the date at which your DAG starts being scheduled. This date can be in the past or in the future. Think of the start date as the start of the data interval you want to process. For example, the 01/01/2021 00:00. In addition to the start date, you need a schedule interval.
The execution time in Airflow is not the actual run time, but rather the start timestamp of its schedule period. For example, the execution time of the first DAG run is 2019–12–05 7:00:00, though it is executed on 2019–12–06.
You can pass parameters from the CLI using --conf '{"key":"value"}' and then use it in the DAG file as "{{ dag_run. conf["key"] }}" in templated field. Save this answer.
{{ dag_run.start_date }}
provides the actual start time of the dag
This is an old question, but I am answering it because the accepted answer did not work for me. {{ dag_run.start_date }}
changes if the DAG run fails and some tasks are retried.
The solution was to use: {{ dag_run.get_task_instance('start').start_date }}
which uses the start date of the first task (DummyOperator task with task_id: start
).
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With