Tensorflow Will Not Import Due to libcublas Issue

Question

I am working on setting up a local dev environment for Deep Learning. I was following the instructions in the first post on this thread on the Fast.ai forum:

http://forums.fast.ai/t/py3-and-tensorflow-setup/1460

Running pip install git+git://github.com/fchollet/keras.git seems to have successfully installed Keras, but with some warnings.

Successfully built Keras
distributed 1.221.8 require msgpack, which is not installed.
tensorboard 1.8.0 has requirement bleach==1.50, but you'll have to bleach 2.1.3 which is incompatible
Tensorboard 1.8.0 has requirement html5lib--0.9999999, but youll have html5lib 1.0.1 which is incompatible.
…
Successfully installed Keras-2.1.6

When I try to import tensorflow, from iPython I get the following stack trace:

In [1]: import tensorflow
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
~/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py in <module>()
     57 
---> 58   from tensorflow.python.pywrap_tensorflow_internal import *
     59   from tensorflow.python.pywrap_tensorflow_internal import __version__

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py in <module>()
     27             return _mod
---> 28     _pywrap_tensorflow_internal = swig_import_helper()
     29     del swig_import_helper

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py in swig_import_helper()
     23             try:
---> 24                 _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
     25             finally:

~/anaconda3/lib/python3.6/imp.py in load_module(name, file, filename, details)
    242         else:
--> 243             return load_dynamic(name, filename, file)
    244     elif type_ == PKG_DIRECTORY:

~/anaconda3/lib/python3.6/imp.py in load_dynamic(name, path, file)
    342             name=name, loader=loader, origin=path)
--> 343         return _load(spec)
    344 

ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-1-d6579f534729> in <module>()
----> 1 import tensorflow

~/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py in <module>()
     22 
     23 # pylint: disable=g-bad-import-order
---> 24 from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
     25 # pylint: disable=wildcard-import
     26 from tensorflow.tools.api.generator.api import *  # pylint: disable=redefined-builtin

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/__init__.py in <module>()
     47 import numpy as np
     48 
---> 49 from tensorflow.python import pywrap_tensorflow
     50 
     51 # Protocol buffers

~/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py in <module>()
     72 for some common reasons and solutions.  Include the entire stack trace
     73 above this error message when asking for help.""" % traceback.format_exc()
---> 74   raise ImportError(msg)
     75 
     76 # pylint: enable=wildcard-import,g-import-not-at-top,unused-import,line-too-long

ImportError: Traceback (most recent call last):
  File "/home/usrname/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/usrname/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/usrname/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/home/usrname/anaconda3/lib/python3.6/imp.py", line 243, in load_module
    return load_dynamic(name, filename, file)
  File "/home/usrname/anaconda3/lib/python3.6/imp.py", line 343, in load_dynamic
    return _load(spec)
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

My system is running Ubuntu 16.04 with a GTX 1060 GPU.

Update: I made the changes to my ~/.bashrc file. So that my LD_LIBRARY_PATH points to the location of lib64, which on my system resides at:

export LD_LIBRARY_PATH=/usr/local/cuda/lib64

Saved the file, then restarted the system and once again tried to import tensorflow from inside iPython. Once again I get an error message containing:

ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

When I run nvidia-smi, the result is :

+-----------------------------------------------------------------------------

+
| NVIDIA-SMI 390.48                 Driver Version: 390.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 00000000:01:00.0  On |                  N/A |
|  8%   54C    P0    26W / 120W |    318MiB /  6075MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1017      G   /usr/lib/xorg/Xorg                           159MiB |
|    0      1826      G   compiz                                       156MiB |
+-----------------------------------------------------------------------------+

When I run apt-update, nvidia, which is the last segment to run returns the following:

Ign:23 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
Hit:24 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release
Get:25 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg [801 B]
Ign:25 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg
Fetched 393 kB in 2min 0s (3,266 B/s)
Reading package lists... Done
W: GPG error: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80
W: The repository 'http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Seems like the public keys are part of the issue. I have tried running the following command:

sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub

The results is:

Executing: /tmp/tmp.GlsZMuriaF/gpg.1.sh --fetch-keys
http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub
gpg: keyserver timed out
gpg: WARNING: unable to fetch URI http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub: keyserver error
ub

Answer 1

This is discussed in this thread in detail. The following worked for me:

1. Download the Cuda 9.0 from Nvida archive

2. Installation Instructions:

   sudo dpkg -i cuda-repo-ubuntu1604-9-0-local_9.0.176-1_amd64.deb
   sudo apt-key add /var/cuda-repo-9-0-local/7fa2af80.pub
   sudo apt-get update
   sudo apt-get install cuda
   

3. Update the paths

   export PATH=/usr/local/cuda-9.0/bin${PATH:+:${PATH}}
   export LD_LIBRARY_PATH=/usr/local/cuda9.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}