Menu
I can't seem to find a way to put descriptions about the Airflow tasks so that they show up in the Dashboard. I am reading their documentation but can't find there either. Does anyone know if this is possible?
583
asked Jul 23 '19 15:07
Airflow's dynamic task mapping feature is built off of the MapReduce programming model. The map procedure takes a set of inputs and creates a single task for each one. The reduce procedure, which is optional, allows a task to operate on the collected output of a mapped task.
A Task is the basic unit of execution in Airflow. Tasks are arranged into DAGs, and then have upstream and downstream dependencies set between them into order to express the order they should run in.
Dynamic DAGs with globals() You can dynamically generate DAGs by working with globals() . As long as a DAG object in globals() is created, Airflow will load it.
Tasks A Task is the basic unit of execution in Airflow. Tasks are arranged into DAGs, and then have upstream and downstream dependencies set between them into order to express the order they should run in. There are three basic kinds of Task:
Airflow detects two kinds of task/process mismatch: Zombie tasks are tasks that are supposed to be running but suddenly died (e.g. their process was killed, or the machine died). Airflow will find these periodically, clean them up, and either fail or retry the task depending on its settings.
The workflow is built with Apache Airflow’s DAG (Directed Acyclic Graph), which has nodes and connectors. A Dependency Tree is created by connecting nodes with connectors. Dynamic Integration: Airflow generates dynamic pipelines using Python as the backend programming language.
Airflow will detect them on a regular basis, clear them up, and then either fail or retry the task, depending on the parameters. Undead Tasks are tasks that are intended to be running but aren’t, which is frequently the result of manually editing Task Instances via the UI. Periodically, airflow will locate them and extinguish them.
You can document both DAGs and tasks with either doc or doc_<json|yaml|md|rst> fields depending on how you want it formatted. These will show up on the dashboard under "Graph View" for DAGs and "Task Details" for tasks.
Example:
"""
# Foo
Hello, these are DAG docs.
"""
...
dag = DAG(
'test.foo',
default_args=default_args,
)
dag.doc_md = __doc__
with dag:
task1 = DummyOperator(
task_id='task1',
)
task1.doc_md = 'Hi, these are task docs.'
Which will result the following:
This feature is documented in https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#dag-task-documentation.
159
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
