Failed to compile cuda_ndarray.cu: libcublas.so.7.5: cannot open shared object file

Question

I am trying to import theano library in an aws instance to use GPU. I have written a python script using boto to automate aws setup which will essentially do an ssh to the instance from my local machine and then start a bash script where I do "python -c 'import theano'" to start the GPU. But I get the following error:

ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: libcublas.so.7.5: cannot open shared object file: No such file or directory

When I tried to import theano module directly in the instance command shell it automatically starts using GPU.

Using gpu device 0: GRID K520 (CNMeM is disabled)

I guess I am missing some other import that has to made while importing through my automation python script. What could possibly be the solution?

Answer 1

I will try to solve this problem clearly and concise, as I found not really good answer for people which are starting using unix or are not familiar with compilation and linking.

The problem has to do with dynamic linking and it can be solved in two ways. First one is by setting LD_LIBRARY_PATH enviroment variable. Assuming cuda is installed in /usr/local/cuda/, just add in your enviroment file /etc/enviroment:

LD_LIBRARY_PATH=/usr/local/cuda/

Or simply in your bashrc:

export LD_LIBRARY_PATH=/usr/local/cuda/lib64/

This solution is not recommended by unix gurus (i am not one i have just read that on the internet and i follow linux gurus). So the solution I found is simple, modify the path where the linux ld software search for libraries by default. To do that just do (you have to do it as root):

cd /etc/ld.so.conf.d/

Then pick for example and edit:

vi libc.conf 

Inside this file just add the path to the lib64 root like:

/usr/local/cuda/lib64/

You would get something like this in the file:

\# libc default configuration

/usr/local/lib

/usr/local/cuda/lib64/

And then just run:

sudo ldconfig

Hope this answer helps people which are starting seen programming, or using high level languages such as python that uses C code below (like theano does) and are not familiar with compilation, linkig...

Answer 2

I faced the same error on Ubuntu 16.04 with cuda 7.5 and found the solution here:

1. cuda 7.5 don't support the default g++ version. Install an supported version and make it the default:

   sudo apt-get install g++-4.9
   
   sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 20
   sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 10
   
   sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.9 20
   sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 10
   
   sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc 30
   sudo update-alternatives --set cc /usr/bin/gcc
   
   sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++ 30
   sudo update-alternatives --set c++ /usr/bin/g++
   

2. Work around a glibc bug - create .theanorc in the home directory with the following settings:

   [global]
   device=gpu
   floatX=float32
   
   [nvcc]
   flags=-D_FORCE_INLINES
   

And don't forget to check environment variables: PATH should contain your cuda bin folder location and CUDA_HOME should contain cuda home location

I've added it to mine .bashrc file this way:

export PATH="/usr/local/cuda/bin:$PATH"
export CUDA_HOME="/usr/local/cuda:$CUDA_HOME"