Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment

I am trying to run Spark using yarn and I am running into this error:

Exception in thread "main" java.lang.Exception: When running with master 'yarn' either HADOOP_CONF_DIR or YARN_CONF_DIR must be set in the environment.

I am not sure where the "environment" is (what specific file?). I tried using:

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

in the bash_profile, but this doesn't seem to help.

like image 413
user1072337 Avatar asked Aug 16 '17 00:08

user1072337


People also ask

Where is Hadoop_conf_dir?

Also, CDH cluster's HADOOP_CONF_DIR should by default be set to /etc/hadoop/conf .


1 Answers

While running spark using Yarn, you need to add following line in to spark-env.sh

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

Note: check $HADOOP_HOME/etc/hadoop is correct one in your environment. And spark-env.sh contains export of HADOOP_HOME as well.

like image 100
Shubhangi Avatar answered Oct 05 '22 16:10

Shubhangi