Apache Spark Native Libraries

Question

I was recently able to build Apache Hadoop 2.5.1 with native 64 bit support. So, I got rid of the annoying Native Libraries Warning.

I'm trying to configure Apache Spark. When I start spark-shell, the same warning appears:

14/09/14 18:48:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Some tips:

I had to download a pre-built 2.4 version of Spark because there is still no profile for Hadoop 2.5 with Maven.

The following exports were added to spark-env.sh:

export HADOOP_CONF_DIR=/opt/hadoop-2.5.1/etc/hadoop

export SPARK_LIBRARY_PATH=/opt/hadoop-2.5.1/lib/native

Didn't work with spark-shell and spark-submit. My Hadoop local installation is configured as pseudo-distributed (ResourceManager + YARN support).

Answer 1

You should add HADOOP_HOME/lib/native to the LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native