Menu
i have searched many links but didn't find any solution to the problem i have. I have seen option to pass key/var into airflow UI ,but it is really confusing for end user to work as which key is associated with which dag. Is there any way to implement functionality like :
While running an airflow job, end user will be asked for values to some parameters and after entering those details airflow will run the job.
750
Adding Params to a DAG To add Params to a DAG , initialize it with the params kwarg. Use a dictionary that maps Param names to a either a Param or an object indicating the parameter's default value.
The task_id is passed to the PythonOperator object. Pass the name of the Python function to the python_callable and the arguments using op_kwargs parameter as dictionary and lastly, the DAG object.
Unfortunately, it's not possible to wait for user input let say in Airflow UI. DAG's are programmatically authored which means defined as a code and they should not be dynamic since they are imported in web server, scheduler and workers in same time and has to be same.
There are two workarounds I came up with, and we use first in production for a while.
1) Create a small wrapper around Variables. For each DAG then load Variables and compose arguments which are then passed into Operators via default_arguments.
2) Add Slack operator which can be programmatically configured to wait for user input. Afterwards, propagate that information via XCOM into next Operator.
57
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
