I just followed the steps in the Cloudera Document, and had GPL Extras Parcel installed on the cluster as well as configured HDFS service via Cloudera Manager. But an error ocurred when trying to read .lzo files on HDFS:
$hadoop fs -text /tmp/Lzo/log.txt.lzo
INFO lzo.GPLNativeCodecLoader: Loaded native gpl library
WARN lzo.LzoCompressor: java.lang.NoSuchFiledError: lzoCompressLevelFunc
ERROR lzo.LzoCodec: Failed to load/initialize native-lzo library
-text: Fatal internal error
java.lang.RuntimeException: native-lzo library not available
I've read a dozen of posts, and known that it's caused by a failure on JNI loading lzo library, but none of them could properly solve my problem. Following are the efforts I've made:
1.All datanodes had installed lzop.
2.JAVA_LIBRARY_PATH in mapred-site.xml was set to /opt/cloudera/parcels/CDH/lib/hadoop/lib/native, which contains liblzo2.* files.
3.HADOOP_CLASSPATH was set to /usr/local/lib which contains hadoop-lzo.jar files.
What else can I do? Any suggestions would be appreciated!
Problem solved! It's caused by java.lang.NoSuchFiledError. Hadoop-lzo-0.4.15 do not have the field lzoCompressLevelFunc, and when I switched to hadoop-lzo0.4.20 the WARN and the ERROR were gone.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With