tensorflow on GPU: no known devices, despite cuda's deviceQuery returning a "PASS" result

Question

From the log output, it looks like you are running the CPU version of TensorFlow (PyPI: tensorflow), and not the GPU version (PyPI: tensorflow-gpu). Running the GPU version would either log information about the CUDA libraries, or an error if it failed to load them or open the driver.

If you run the following commands, you should be able to use the GPU in subsequent runs:

$ pip uninstall tensorflow
$ pip install tensorflow-gpu

None of the other answers here worked for me. After a bit of tinkering I found that this fixed my issues when dealing with Tensorflow built from binary:

---

Step 0: Uninstall protobuf

pip uninstall protobuf

Step 1: Uninstall tensorflow

pip uninstall tensorflow
pip uninstall tensorflow-gpu

Step 2: Force reinstall Tensorflow with GPU support

pip install --upgrade --force-reinstall tensorflow-gpu

Step 3: If you haven't already, set CUDA_VISIBLE_DEVICES

So for me with 2 GPUs it would be

export CUDA_VISIBLE_DEVICES=0,1

In my case:

pip3 uninstall tensorflow

is not enough. Because when reinstall with:

pip3 install tensorflow-gpu

It is still reinstall tensorflow with cpu not gpu. So, before install tensorflow-gpu, I tried to remove all related tensor folders in site-packages uninstall protobuf, and it works!

For conclusion:

pip3 uninstall tensorflow

Remove all tensor folders in ~\Python35\Lib\site-packages

pip3 uninstall protobuf
pip3 install tensorflow-gpu

Might seem dumb but a sudo reboot has fixed the exact same problem for me and a couple others.

The answer that saved my day came from Mark Sonn. Simply add this to .bashrc and source ~/.bashrc if you are on Linux:

export CUDA_VISIBLE_DEVICES=0,1

Previously I had to use this workaround to get tensorflow recognize my GPU:

import tensorflow as tf

gpus = tf.config.experimental.list_physical_devices(device_type="GPU")
tf.config.experimental.set_visible_devices(devices=gpus[0], device_type="GPU")
tf.config.experimental.set_memory_growth(device=gpus[0], enable=True)

Even though the code still worked, adding these lines every time is clearly not something I would want. My version of tensorflow was built from source according to the documentation to get v2.3 support CUDA 10.2 and cudnn 7.6.5.

If anyone having trouble with that, I suggest doing a quick skim over the docs. Took 1.5 hours to build with bazel. Make sure you have gcc7 and bazel installed.