Why is homebrew Hadoop 2.3 not working OSX Mavericks?

Question

I am running into the following issues after using homebrew to install hadoop. I followed the guide here:

http://glebche.appspot.com/static/hadoop-ecosystem/hadoop-hive-tutorial.html

Setting the following environment variables in bashrc:

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_55.jdk/Contents/Home
export HADOOP_INSTALL=/usr/local/Cellar/hadoop/2.3.0
export HADOOP_HOME=$HADOOP_INSTALL
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL

After running a hadoop namenode -format.. I attempt to run start-dfs.sh and get the following:

14/05/05 21:19:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: set hadoop variables
localhost: starting namenode, logging to /usr/local/Cellar/hadoop/2.3.0/libexec/logs/mynotebook.local.out
localhost: Error: Could not find or load main class org.apache.hadoop.hdfs.server.namenode.NameNode
localhost: set hadoop variables
localhost: starting datanode, logging to /usr/local/Cellar/hadoop/2.3.0/libexec/logs/mynotebook.local.out
localhost: Error: Could not find or load main class org.apache.hadoop.hdfs.server.datanode.DataNode
Starting secondary namenodes [0.0.0.0]
0.0.0.0: set hadoop variables
0.0.0.0: secondarynamenode running as process 12747. Stop it first.
14/05/05 21:19:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

How to I get around this issue?

Answer 1

Based on the first line of the second message,

"14/05/05 21:19:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable"

I suppose that you're running hadoop in a 64 bit operating system. Hadoop is built from default in a 32 bit system, I had the same issue and the same message. What you have to do is re-build hadoop from the source on your system.

I suggest you to use the guide below, it's for the 2.2 version but it's ok for the 2.3 version too

http://csrdu.org/nauman/2014/01/23/geting-started-with-hadoop-2-2-0-building/

Or the official guide

http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/NativeLibraries.html#Build