ImportError: libnvidia-fatbinaryloader.so.375.39: cannot open shared object file: No such file or directory

Question

I'm using Ubuntu 16.04, Cuda 8.0 and cudann-v5.1. I uninstalled Tensorflow-CPU version and reinstalled tensorflow-GPU enabled. Followed the instructions given here: https://alliseesolutions.wordpress.com/2016/09/08/install-gpu-tensorflow-from-sources-w-ubuntu-16-04-and-cuda-8-0-rc/

However, when I try to load tensorflow, I get the following error:

>>> import tensorflow as tf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/__init__.py", line 51, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 56, in <module>
    raise ImportError(msg)
ImportError: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
ImportError: libnvidia-fatbinaryloader.so.375.39: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

Answer 1

I encountered this issue as well, there were two issues that needed to be resolved.

1. I added /usr/lib/nvidia-375 to my LD_LIBRARY_PATH environment variable. You can verify that the file libnvidia-fatbinaryloader.so.375.39 lives in that directory. If not, find where it does live and add that path. It's not clear to me why this wasn't picked up properly in compiling the sources.

2. Next I encountered the error:

   libstdc++.so.6: version `CXXABI_1.3.8' not found
   

If you encounter that it's because you have a newer version of gcc than is available in anaconda or your python installation. For me that meant adding this path to LD_LIBRARY_PATH: /usr/lib/x86_64-linux-gnu/libstdc++.so.6

I also had to rename the old libstdc++.so.6 at the path shown in the error message. I couldn't find a way to convince python not to look in the default path without just renaming the file. There may be a cleaner way to do this, but that worked for me.

There were a lot of hidden gotchas in the installation.

Answer 2

In my case tensorflow-gpu was installed and working for a time, but installation of nvidia-opencl-icd-384 caused apt to upgrade my nvidia drivers from 384.47 to 384.59 but did NOT upgrade libcuda, nvidia-settings, and perhaps other packages. Upgrading libcuda should most likely solve the issue but probably best to purge and reinstall.

sudo apt-get purge nvidia*
sudo apt-get install libcuda1-384 nvidia-384 nvidia-384-dev nvidia-prime nvidia-settings

For me, symbolic link for libnvidia-fatbinaryloader.so.384.37 -> libnvidia-fatbinaryloader.so.384.59 in /usr/lib and /usr/lib32 did not correct the issue and LD_LIBRARY_PATH was already correctly configured.