Ubuntu 16.04, CUDA 8 - CUDA driver version is insufficient for CUDA runtime version

Question

I've installed the latest nvidia drivers (375.26) manually, and installed CUDA using cuda_8.0.44_linux.run (skipping the driver install there, since the bundled drivers are older, 367 I think).

Running the deviceQuery in CUDA samples produces the following error however:

~/cudasamples/NVIDIA_CUDA-8.0_Samples/1_Utilities/deviceQuery$ ./deviceQuery 
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL

Version info:

$ nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Sun_Sep__4_22:14:01_CDT_2016
Cuda compilation tools, release 8.0, V8.0.44

$ nvidia-smi
Sat Dec 31 17:25:03 2016       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.26                 Driver Version: 375.26                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 0000:01:00.0      On |                  N/A |
|  0%   39C    P8    11W / 230W |    464MiB /  8110MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0       974    G   /usr/lib/xorg/Xorg                             193MiB |
|    0      1816    G   compiz                                         172MiB |
|    0      2178    G   ...ignDownloads/Enabled/MaterialDesignUserMa    96MiB |
+-----------------------------------------------------------------------------+

$  cat /proc/driver/nvidia/version 
NVRM version: NVIDIA UNIX x86_64 Kernel Module  375.26  Thu Dec  8 18:36:43 PST 2016
GCC version:  gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 

The anwer to similar problems has been updating the nvidia display drivers, though in my case this is already done. Does anyone have any ideas? Thanks.

Answer 1

Running

sudo apt-get purge nvidia-*

and reinstalling the drivers using

sudo apt-get install nvidia-375

solved it. Just for the record, the first time I updated the drivers using the GUI (Additional Drivers tab in Software & Updates).

Answer 2

First, check "CUDA Toolkit and Compatible Driver Versions" from here, and make sure that your cuda toolkit version is compatible with your cuda-driver version， e.g. if your driver version is nvidia-390, your cuda version must lower than CUDA 9.1.
Then, back to this issue. This issue is caused by "your cuda-driver version doesn't match your cuda version, and your CUDA local version may also different from the CUDA runtime version(cuda version in some specific virtual environments)."
I met the same issue when I tried to run tensorflow-gpu under the environment of "tensorflow_gpuenv" created by conda, and tried to test whether the "gpu:0" device worked. My driver version is nvidia-390 and I've already install cuda 9.0, so it doesn't make sense that raising that weird issue. I finally found the reason that the cuda version in the conda virtual environment is cuda 9.2 which isn't compatible with nvidia-390. I solved the issue by following steps in ubuntu 18.04:

• check cuda driver version
  ~$ nvidia-smi or ~$ cat /proc/driver/nvidia/version
• check local cuda version
  ~$ nvcc --version or ~$ cat /usr/local/cuda/version.txt

• check local cudnn version
  ~$ cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2

  ---

• check cuda version in virtual environment
  ~$ conda list you can see something like these :
  cudatoolkit      9.2       0
  cudnn        7.3.1      cuda9.2_0
  you may find that the cuda version in virtual environment is different from the local cuda version, and isn't compatible with driver version nvidia-390.

So reinstall cuda in the virtual environment:

• reinstall cuda : ~$ conda install cudatoolkit=8.0
  (change the version number '8.0' to other version number which match your driver version, and your cudnn version will update automatically to match the new version cuda )

Answer 3

I have followed the instructions on this page, and it works for me.

https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=debnetwork

First, download installer for Linux Ubuntu 16.04 x86_64.

Next, follow these steps to install Linux Ubuntu:

1. sudo dpkg -i cuda-repo-ubuntu1604_9.2.148-1_amd64.deb

2. sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub

3. sudo apt-get update

4. sudo apt-get install cuda

Answer 4

I was with the same problem. I had the version nvidia-390 installed on Ubuntu 18.04.2 LTS. My Graphic card is GeForce GTX 1080, and using tensorflow 1.12.0. I successfully solved this problem by removing the old version:

sudo apt-get purge nvidia-*

And then installing the version 418

sudo apt-get install nvidia-driver-418 nvidia-settings