Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

Airflow: Creating a DAG in airflow via UI

Tags:

Airflow veterans please help,

I was looking for a cron replacement and came across apache airflow.

We have a setup where multiple users should be able to create their own DAGs and schedule their jobs.

Our users are a mix of people who may not know how to write the DAG python file. Also, they may not have access to the server where airflow is running.

Is it possible to create an airflow DAG via UI. I could not find any reference to the same. All examples speak about creating a python file and uploading it to the $AIRFLOW_HOME/dag/ directory. Users will not have access to this directory.

Rundeck for example allows user to add workflows and task dependencies via UI. Is there a plugin/functionality similar to this in airflow.

PS: I really like the way airflow shows the dependency graphs and want to try it out. But If creating a DAG is so complicated, then it will be a major problem for lots of my end users.

like image 654
Mukul Jain Avatar asked Feb 26 '18 10:02

Mukul Jain

People also ask

Can we create DAG from Airflow UI?

Because everything in Airflow is code, you can dynamically generate DAGs using Python alone. As long as a DAG object in globals() is created by Python code that lives in the dags_folder , Airflow will load it. In this guide, we'll cover a few of the many ways you can generate DAGs.

How do you deploy DAG in Airflow?

When your new DAG file is loaded in Airflow you can recognize it in the UI thanks to the version number. Because your DAG file name = DAG Id you could even improve the deployment script by adding some Airflow command line to automatically switch ON your new DAGs once they are deployed.

2 Answers

I don't think there is an out of the box solution. What I would try with your task is the combination: CWL(Common Workflow Language), Airflow, Rabix Composer

So, CWL is just a specification, you describe your workflow in YAML like syntax.

There is a package pip3 install cwl-airflow it can parse CWL into Airflow DAG https://github.com/Barski-lab/cwl-airflow.

Rabix Composer UI/graphical CWL editor.

like image 147
Andrey Kartashov Avatar answered Nov 12 '22 07:11

Andrey Kartashov

elyra seems to be able to create airflow pipelines (did not try it):

https://elyra.readthedocs.io/en/latest/getting_started/overview.html

enter image description here

like image 39
Stefan Avatar answered Nov 12 '22 05:11

Stefan
Sign in to Comment

Related questions

Variables marked as const using structured bindings are not const
Spring Retry with Transactional
What are the use cases of the newly proposed Pin type?
How do I implement an async I/O bound operation from scratch?
Using React forwardRef with Typescript generic JSX arguments
COUNT(id) or MAX(id) - which is faster?
Using `is` operator with value type tuples gives error
If Spring can successfully intercept intra class function calls in a @Configuration class, why does it not support it in a regular bean?
Apache Spark + Delta Lake concepts
Why does sync.Mutex largely drop performance when goroutine contention is more than 3400?
SwiftUI: Pop to root view when selected tab is tapped again
Flutter- FCM A background message could not be handled in Dart as no onBackgroundMessage handler has been registered

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!