Menu
I want the tasks in the DAG to all finish before the 1st task of the next run gets executed.
I have max_active_runs = 1, but this still happens.
default_args = { 'depends_on_past': True, 'wait_for_downstream': True, 'max_active_runs': 1, 'start_date': datetime(2018, 03, 04), 'owner': 't.n', 'email': ['[email protected]'], 'email_on_failure': True, 'email_on_retry': False, 'retries': 3, 'retry_delay': timedelta(minutes=4) } dag = DAG('example', default_args=default_args, schedule_interval = schedule_interval) (All of my tasks are dependent on the previous task. Airflow version is 1.8.0)
Thank you
772
If you stop a DAG and clear the task from the UI, the running tasks in the executor will not stop. When the task is in a running state, you can click on CLEAR, and it will call job. kill() function on the task. This function will set the task's status to shut_down, which will be shifted to up_for_retry immediately.
By default, the Celery executor will run a maximum of 16 tasks concurrently.
You can run a task independently by using -i/-I/-A flags along with the run command.
In order to filter DAGs (e.g by team), you can add tags in each dag. The filter is saved in a cookie and can be reset by the reset button.
I changed to put max_active_runs as an argument of DAG() instead of in default_arguments, and it worked.
Thanks SimonD for giving me the idea, though not directly pointing to it in your answer.
169
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
