Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Airflow - Get start time of dag run

Is it possible to get the actual start time of a dag in Airflow? By start time I mean the exact time the first task of a dag starts running.

I know I can use macros to get the execution date. If the job is ran using trigger_dag this is what I would call a start time but if the job is ran on a daily schedule then {{ execution_date }} returns yesterdays date.

I have also tried to place datetime.now().isoformat() in the body of the dag code and then pass it to a task but this seems to return the time the task is first called rather than when the dag itself started.

like image 849
RoachLord Avatar asked Nov 03 '17 13:11

RoachLord


People also ask

How do you get the DAG run date in Airflow?

Your DAG will be instantiated for each schedule along with a corresponding DAG Run entry in the database backend. If you run a DAG on a schedule_interval of one day, the run stamped 2020-01-01 will be triggered soon after 2020-01-01T23:59. In other words, the job instance is started once the period it covers has ended.

What is start date in Airflow DAG?

The start date is the date at which your DAG starts being scheduled. This date can be in the past or in the future. Think of the start date as the start of the data interval you want to process. For example, the 01/01/2021 00:00. In addition to the start date, you need a schedule interval.

What is execution time in Airflow?

The execution time in Airflow is not the actual run time, but rather the start timestamp of its schedule period. For example, the execution time of the first DAG run is 2019–12–05 7:00:00, though it is executed on 2019–12–06.

How do you pass runtime arguments in Airflow DAG?

You can pass parameters from the CLI using --conf '{"key":"value"}' and then use it in the DAG file as "{{ dag_run. conf["key"] }}" in templated field. Save this answer.


2 Answers

{{ dag_run.start_date }} provides the actual start time of the dag

like image 77
RoachLord Avatar answered Sep 22 '22 18:09

RoachLord


This is an old question, but I am answering it because the accepted answer did not work for me. {{ dag_run.start_date }} changes if the DAG run fails and some tasks are retried.

The solution was to use: {{ dag_run.get_task_instance('start').start_date }} which uses the start date of the first task (DummyOperator task with task_id: start).

like image 37
Oscar Perez Avatar answered Sep 22 '22 18:09

Oscar Perez