How to pass Docker CLI `--gpus` Options in Kubernetes or enable GPU support without installing `nvidia-docker2` (Docker 19.03)

Question

I'm currently using Docker 19.03 and Kubernetes 1.13.5 and Rancher 2.2.4. Since 19.03, Docker has officially support natively NVIDIA GPUs just by passing --gpus option. Example (from NVIDIA/nvidia-docker github):

 docker run --gpus all nvidia/cuda nvidia-smi

But in Kubernetes, there's no option to pass Docker CLI options. So if I need to run a GPU instance, I have to install nvidia-docker2, which is not convenient to use.

Is there anyway to pass the Docker CLI options or passing NVIDIA runtime without installing nvidia-docker2

Answer 1

GPU's are scheduled via device plugins in Kubernetes.

The official NVIDIA GPU device plugin has the following requirements:

• Kubernetes nodes have to be pre-installed with NVIDIA drivers.
• Kubernetes nodes have to be pre-installed with nvidia-docker 2.0
• nvidia-container-runtime must be configured as the default runtime for docker instead of runc.
• NVIDIA drivers ~= 361.93

Once the nodes are setup GPU's become another resource in your spec like cpu or memory.

spec:
  containers:
  - name: gpu-thing
    image: whatever
    resources:
      limits:
        nvidia.com/gpu: 1