Rapids / docker: could not select device driver "" with capabilities: [[gpu]]

Question

I'm new to Rapids, and rarely have had a good experience with conda. So I'm trying to work with a containerized version. I'm new to Docker, and the combination of unknowns leaves me unable to sort things out.

I have an Ubuntu 18.04 server,

# uname -v
#30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020

on which I installed a fresh version of Docker

# apt-get install docker docker-ce docker-ce-cli containerd.io
# docker --version
Docker version 19.03.8, build afacb8b7f0

This machine has cuda v10.2 installed

# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

and Python v3.6.9

# python3 --version
Python 3.6.9

As shown in the NVIDIA Container Toolkit Quickstart section, I install the nvidia-docker list to /etc/apt/sources.list.d/

# curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
# curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu18.04/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

explicitly substituting ubuntu18.04 for $distribution, since that is the Ubuntu equivalent for Linux Mint 19.3.

Following the Start Container and Notebook Server instructions in RAPIDS - Open GPU Data Science, I pulled the 0.13-cuda10.2-runtime-ubuntu18.04-py3.6 runtime.

# docker pull rapidsai/rapidsai:0.13-cuda10.2-runtime-ubuntu18.04-py3.6

A long time, and several GB later, all seemed to be OK. (No warnings or error messages.) Furthermore, it looks like the image was registered with Docker.

# docker images -a
REPOSITORY          TAG                                       IMAGE ID            CREATED             SIZE
rapidsai/rapidsai   0.13-cuda10.2-runtime-ubuntu18.04-py3.6   c7440af853b5        4 days ago          9.26GB
rapidsai/rapidsai   cuda10.2-runtime-ubuntu18.04-py3.6        c7440af853b5        4 days ago          9.26GB

However, I next tried to start up the notebook server:

# docker run --gpus all --rm -it -p 8888:8888 -p 8787:8787 -p 8786:8786 \
       rapidsai/rapidsai:cuda10.0-runtime-ubuntu18.04-py3.6
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

This seems surprising, as there are two GTX 1080 Ti GPUs detected

# nvidia-smi
Fri May  8 16:41:57 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  On   | 00000000:08:00.0 Off |                  N/A |
| 21%   38C    P8    10W / 250W |      1MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  On   | 00000000:42:00.0 Off |                  N/A |
| 23%   42C    P8    10W / 250W |      1MiB / 11177MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

After cleaning things up

# docker system prune -a
# apt-get purge docker docker-engine docker.io containerd runc    

I re-installed docker and pulled the rapidsai image again. The result was unchanged.

Is there a conflict with the NVIDIA Driver Version: 440.33.01?

Any suggestions?

Answer 1

Thanks for trying out RAPIDS. Did you happen to install nvidia-container-toolkit? https://github.com/NVIDIA/nvidia-docker#quickstart. I didn't see that in your steps and missing it could cause that issue. It's in our prerequisites on https://rapids.ai/start.html