Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to set up emr_default in airflow

Tags:

python

airflow

Dear Stackoverflow member,

I am trying to use Airflow to schedule a DAG that launch a cluster, run some spark jobs then terminated itself. I tried to follow this example: https://insight.io/github.com/apache/incubator-airflow/blob/master/airflow/contrib/example_dags/example_emr_job_flow_automatic_steps.py

It gave me the error: airflow.exceptions.AirflowException: The conn_id emr_default isn't defined

I probably need to have emr_default set up (please correct me if this is not the case). Does anyone have try this before and give me some hints on how to set it up? Airflow website doesn't provide too much on emr connection: http://pythonhosted.org/airflow/configuration.html?highlight=connection#connections

Thanks!

like image 938
Chengzhi Avatar asked Oct 30 '22 06:10

Chengzhi


2 Answers

On the airflow UI, go to Admin -> Connections tab, and edit the values for emr_default row.

like image 167
Him Avatar answered Nov 09 '22 11:11

Him


As@Him said, you can edit in airflow. but if you can't find emr_default, you can go to Admin -> Connections -> Create to add any connection you want

like image 39
zhongjiajie Avatar answered Nov 09 '22 10:11

zhongjiajie