Where is the classpath set for hadoop

Question

Where is the classpath for hadoop set? When I run the below command it gives me the classpath. Where is the classpath set?

  bin/hadoop classpath

I'm using hadoop 2.6.0

Answer 1

As said by almas shaikh it's set in hadoop-config.sh, but you could add more jars to it in hadoop-env.sh

Here is a relevant code from hadoop-env.sh which adds additional jars like capacity-scheduler and aws jar's.

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

# Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
  if [ "$HADOOP_CLASSPATH" ]; then
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
  else
    export HADOOP_CLASSPATH=$f
  fi
done

# ... some other lines omitted

# Add Aws jar
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:share/hadoop/tools/lib/*

Answer 2

Open your bash profile (~/.profile or ~/.bash_profile) for editing and add the following:

1. export HADOOP_HOME="/usr/local/Cellar/hadoop" then Replace with your own path

2. export HADOOP_CLASSPATH=$(find $HADOOP_HOME -name '*.jar' | xargs echo | tr ' ' ':') Save the changes and reload.

3. source ~/.profile

Answer 3

When you run hadoop command, it sources a file hadoop-config.sh that resides in $HADOOP_HDFS_HOME/libexec which sets your classpath (CLASSPATH) by picking jars residing in various directories viz.

$HADOOP_HDFS_HOME/share/hadoop/mapreduce 
$HADOOP_HDFS_HOME/share/hadoop/common
$HADOOP_HDFS_HOME/share/hadoop/hdfs etc.