Running tensorflow with file on HDFS (cannot find libhdfs.so)

Question

I am running into an error "libhdfs.so: cannot open shared object file: No such file or directory" (stack trace below) while trying to run a python script invoking a Tensorflow reader on a file stored in HDFS. I am running the script on a node on the cluster which has Tensorflow in a virtualenv, activated at the time of execution. I set the following environment variables before execution:

• export HADOOP_HDFS_HOME=$HADOOP_HDFS_HOME:/opt/cloudera/parcels/CDH
• export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
• export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/cloudera/parcels/CDH/lib/libhdfs.so
• export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${JAVA_HOME}/jre/lib/amd64/server

I execute the script as such:

• CLASSPATH=$($LD_LIBRARY_PATH} classpath --glob) python TEST.py

This is the code in the script:

filename_queue = tf.train.string_input_producer([   
   "hdfs://hostname:port/user/hdfs/test.avro" ]) 
reader =
   tf.WholeFileReader() key, value = reader.read(filename_queue)

   with tf.Session() as sess:
       coord = tf.train.Coordinator()
       threads = tf.train.start_queue_runners(coord=coord)
       sess.run([key,value])
       coord.request_stop()
       coord.join(threads)

Below is the stack trace of the error. Any ideas on what is causing this error is appreciated (I have already checked that the LD_LIBRARY_PATH variable has an explicit pointer to the libhdfs.so file prior to execution, unable to figure out why it still cannot find the file).

Traceback (most recent call last):
  File "TEST.py", line 25, in <module>
    sess.run([key,value])
  File "/home/username/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 767, in run
    run_metadata_ptr)
  File "/home/username/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _run
    feed_dict_string, options, run_metadata)
  File "/home/username/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1015, in _do_run
    target_list, options, run_metadata)
  File "/home/username/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1035, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: libhdfs.so: cannot open shared object file: No such file or directory
     [[Node: ReaderReadV2 = ReaderReadV2[_device="/job:localhost/replica:0/task:0/cpu:0"](WholeFileReaderV2, input_producer)]]

Caused by op u'ReaderReadV2', defined at:
  File "TEST.py", line 19, in <module>
    key, value = reader.read(filename_queue)
  File "/home/username/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/io_ops.py", line 272, in read
    return gen_io_ops._reader_read_v2(self._reader_ref, queue_ref, name=name)
  File "/home/username/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 410, in _reader_read_v2
    queue_handle=queue_handle, name=name)
  File "/home/username/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 763, in apply_op
    op_def=op_def)
  File "/home/username/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2395, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/username/tensorflow/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1264, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): libhdfs.so: cannot open shared object file: No such file or directory

Answer 1

I also encountered this issue the solution for me was copying this file to:

$HADOOP_HDFS_HOME/lib/native

If you don't know the location of this file execute the following command to find it's location:

sudo updatedb
locate libhdfs.so

This will give you the location of the file. Next copy the file to $HADOOP_HDFS_HOME/lib/native:

cp locationOflibhdfs.so $HADOOP_HDFS_HOME/lib/native

Note: Replace locationOflibhdfs.so with the location of the libhdfs.so file.