Non-OK-status: GpuLaunchKernel(...) status: Internal: no kernel image is available for execution on the device

Question

I run my code on tensorflow 2.1.0 Anaconda with CUDA Toolkit 10.1 CUDNN 7.6.0 (Windows 10) and it returns a issue

F .\tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: no kernel image is available for execution on the device

My GPU : GT940MX Compute Capability 5.0

I already run the nvcc -V and it returns :

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:26_Pacific_Standard_Time_2019
Cuda compilation tools, release 10.1, V10.1.105

This is the full result :

2020-08-05 10:05:48.368012: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 10:06:00.488544: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-08-05 10:06:48.153611: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce 940MX computeCapability: 5.0
coreClock: 0.8605GHz coreCount: 4 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 37.33GiB/s
2020-08-05 10:06:48.164731: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 10:06:48.245826: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-05 10:06:48.296245: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-05 10:06:48.338860: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-05 10:06:48.439393: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-05 10:06:48.489830: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-05 10:06:48.941872: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-05 10:06:48.946651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-05 10:06:48.951881: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-08-05 10:06:48.979077: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23d29b660d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-05 10:06:48.985680: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-08-05 10:06:48.990616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce 940MX computeCapability: 5.0
coreClock: 0.8605GHz coreCount: 4 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 37.33GiB/s
2020-08-05 10:06:49.003356: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-05 10:06:49.009869: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-05 10:06:49.014858: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-05 10:06:49.020699: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-05 10:06:49.028876: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-05 10:06:49.033607: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-05 10:06:49.039192: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-05 10:06:49.045288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-08-05 10:06:49.218497: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-05 10:06:49.223536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2020-08-05 10:06:49.226857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2020-08-05 10:06:49.230413: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1460 MB memory) -> physical GPU (device: 0, name: GeForce 940MX, pci bus id: 0000:01:00.0, compute capability: 5.0)
2020-08-05 10:06:49.244107: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x23d301b8fa0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-05 10:06:49.250377: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce 940MX, Compute Capability 5.0
2020-08-05 10:06:49.446601: F .\tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: no kernel image is available for execution on the device

What are the issues and how to fix it?

Answer 1

Looks like this is an issue with Python 3.8 and Tensorflow 2.3. I tried the tensorflow 2.3.0 with python 3.7, but it returns an error with python 3.7 because python38.dll (I don't remember exactly the error and i already delete the env), anyway i used python 3.7 on anaconda env and installed tensorflow 2.1.0 with pip and it works.

I posted the question too in github and this question is answered in github https://github.com/tensorflow/tensorflow/issues/42052

Answer 2

As per the screenshot below, Tensorflow Versions 2.1, 2.2 and 2.3 work with cuDNN version of 7.4 but cuDNN version of your GPU is 7.6.

That might be, most probably, the reason for the error.

Solution is to downgrade cuDNN Version of your GPU.

Existing Version of cuDNN can be uninstalled through the Windows Control Panel by using the Programs and Features widget.

New Version of cuDNN can be installed as shown in this NVIDIA Installation Guide.

Also, please refer this Github Issue to know more on how to downgrade cuDNN Version.

The above screenshot has been taken from this Tensorflow Documentation.