How to capture passed --conf parameter in called DAG in Airflow

Question

I am trying to run a DAG from REST API and pass some parameters to it. The DAG should be able to catch the parameters and use it. The problem is I am able to trigger the DAG from REST API,but the DAG is not able to catch the parameters passed. Is there a way to achieve this?

I am triggering the DAG from REST API as below.It passes the parameters in --conf

http://abcairflow.com:8090/admin/rest_api/api?api=trigger_dag\&dag_id=trigger_test_dag\&conf=%7B%22key%22%3A%2

How to capture the values passed in conf value in the called DAG. As far as I know the conf should take the URL encoded JSON format data.

DAG code:`

def run_this_func(**kwargs):
print(kwargs)

run_this = PythonOperator(
    task_id='run_this',
    python_callable=run_this_func,
    dag=dag
)`

Answer 1

The external parameters passed are part of the dag_run object. They can be accessed as follows:

API Request

import requests

headers = {
    'Cache-Control': 'no-cache',
    'Content-Type': 'application/json',
}

data = '{"conf":"{\\"Key1\\":\\"Value1\\"}"}'

response = requests.post('http://localhost:8080/api/experimental/dags/<dag_id>/dag_runs', headers=headers, data=data)

DAG

 def run_this_func(**context):
    print("Received {} for key=message".format(context["dag_run"].conf))


run_this = PythonOperator(task_id="run_this", python_callable=run_this_func, provide_context=True, dag=dag)

Answer 2

I did not know that you could trigger a DAG with HTTP GET, but I've successfully triggered with conf using POST and following the documentation https://airflow.apache.org/api.html

For example triggering the dag "trigger_test_dag":

curl -X POST --data '"conf":"{\"key\":\"value\"}"' \
"http://abcairflow.com:8090/api/experimental/dags/trigger_test_dag/dag_runs"

Pay attention to the escaping of apostrophes as conf needs to be a string. I guess you can do a base 64 encode, and then decode in the DAG, to the string if you prefer that.