How to unload an NVIDIA kernel module 'nvidia' for new driver installation?

Question

I needed to upgrade my nvidia driver so that I have tried running NVIDIA-LInux-x86_64.run file

However, I was seeing following message

ERROR: An NVIDIA kernel module 'nvidia' appears to already be loaded in your kernel.  This may be because it is in use (for example, by an X server, a CUDA program, or the NVIDIA Persistence Daemon), but this may also happen if your kernel was configured without support for module unloading.  Please be sure to exit any programs that may be using the GPU(s) before attempting to upgrade your driver.  If no GPU-based programs are running, you know that your kernel supports module unloading, and you still receive this message, then an error may have occured that has corrupted an NVIDIA kernel module's usage count, for which the simplest remedy is to reboot your computer.

I have already unloaded nvidia-drm and when I tried to unload nvidia

$ sudo modprobe -r nvidia
modprobe: FATAL: Module nvidia is in use.

Can anyone guide me on installing this new driver without any issue?

Thanks

Answer 1

Use lsof /dev/nvidia* to find the processes that are using the old driver. In my case it was "nvidia-persistenced". Just kill the process by pid and retry the installer NVIDIA-***.run

# lsof /dev/nvidia*
COMMAND    PID                USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
nvidia-pe 1334 nvidia-persistenced    2u   CHR 195,255      0t0  420 /dev/nvidiactl
nvidia-pe 1334 nvidia-persistenced    3u   CHR   195,0      0t0  421 /dev/nvidia0
nvidia-pe 1334 nvidia-persistenced    5u   CHR   195,0      0t0  421 /dev/nvidia0
nvidia-pe 1334 nvidia-persistenced    6u   CHR   195,0      0t0  421 /dev/nvidia0
nvidia-pe 1334 nvidia-persistenced    7u   CHR   195,0      0t0  421 /dev/nvidia0

Answer 2

I just removed existing driver and reinstalled