google cloud vm cannot find GPU after restart

Question

I created one google cloud vm with GPU. The GPU was working after instance created. But after restarting the vm, the GPU was gone.

I ran nvidia-smi and got this error:
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

Who knows how to fix it? Thanks

Answer 1

I had the same issue using a Google Deep Learning VM with CUDA 11.3 this solved it:

sudo /opt/deeplearning/install-driver.sh

Not sure is a definitive solution but works for now.

There seems to be some issue with the kernel headers not being found, couldn't figure it out exactly.