docker build with nvidia runtime

Question

I have a GPU application that does unit-testing during the image building stage. With Docker 19.03, one can specify nvidia runtime with docker run --gpus all but I also need access to the gpus for docker build because I do unit-testing. How can I achieve this goal?

For older version of docker that use nvidia-docker2 it was not possible to specifiy runtime during build stage, BUT you can set the default runtime to be nvidia, and docker build works fine that way. Can I do that in Docker 19.03 that doesn't need nvidia-docker anymore? If so, how?

Answer 1

You need use nvidia-container-runtime as explained in docs: "It is also the only way to have GPU access during docker build".

Steps for Ubuntu:

1. Install nvidia-container-runtime:

   sudo apt-get install nvidia-container-runtime

2. Edit/create the /etc/docker/daemon.json with content:

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
         } 
    },
    "default-runtime": "nvidia" 
}

3. Restart docker daemon:

   sudo systemctl restart docker

4. Build your image (now GPU available during build):

   docker build -t my_image_name:latest .

Answer 2

A "solution" I found is to first run a base image with the host nvidia drivers mounted on it

docker run -it --rm --gpus ubuntu

And then build my app within the container manually and commit the resulting image. This is not ideal and it would be best to have access to nvidia-smi during the build phase.