Different nvidia driver versions per docker container

Question

Is it possible to run two Nvidia Docker containers, each with its own Nvidia driver version?

On my cloud instance, I have an older application running for which newer Nvidia drivers are causing issues. I'd like the ability to keep running it with the older driver, while allowing newer applications on the same instance to use newer drivers. I was thinking I could accomplish this with containers but I'm worried that they only allow you to containerize things in user space.

Answer 1

Answering for an update, actually now its possible to different driver version inside different containers. All you have to do is change the NVIDIA Container Toolkit config file (/etc/nvidia-container-runtime/config.toml) so that the root directive points to the driver container as shown below:

disable-require = false
swarm-resource = "DOCKER_RESOURCE_GPU"

[nvidia-container-cli]
root = "/run/nvidia/driver"
path = "/usr/bin/nvidia-container-cli"
environment = []
debug = "/var/log/nvidia-container-toolkit.log"
ldcache = "/etc/ld.so.cache"
load-kmods = true
no-cgroups = false
user = "root:video"
ldconfig = "@/sbin/ldconfig.real"

[nvidia-container-runtime]
debug = "/var/log/nvidia-container-runtime.log"

This command can be directly used to make this modification :

$sudo sed -i 's/^#root/root/' /etc/nvidia-container-runtime/config.toml

After that you can run a container with a required version of driver in background using :

$sudo docker run --name nvidia-driver -d --privileged --pid=host \
  -v /run/nvidia:/run/nvidia:shared \
  -v /var/log:/var/log \
  --restart=unless-stopped \
  nvidia/driver:450.80.02-ubuntu18.04

and then run the GPU conatiner with required version of CUDA for your work:

$sudo docker run --gpus all nvidia/cuda:11.0-base nvidia-smi