I am trying to run tensorflow with gpu support in a docker on a virtual machine. I have tried lots of online solutions including:
none of the solutions work for me, here some steps:
I verified that drivers and cuda and cudnn toolkit are installed inside the container using nvidia-smi and nvcc -V:
Python version is : Python 3.8.10
and tensorflow version is:
import tensorflow as tf
tf.__version__
'2.6.0'
The error appears with: tf.config.list_physical_devices()
So the GPU is somehow not visible to the tensorflow. All tensorflow builds return the same error:
E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NOT_INITIALIZED: initialization error
but for example for 1.14 there is an additional comment regarding the CPU type:
Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
The GPU is a A100 and the CPU is Intel(R) Xeon(R) Gold 6226R.
What is going on here? How do I fix this?
I realized that the GPU has a multi-instance feature:
Therefore, the GPU instances should be configured:
sudo nvidia-smi mig -cgi 0 -C
and afterwards when calling nvidia-smi you get:
And the problem is solved!
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With