Running Keras with double precision fails

Question

I am trying to run LeNet on Keras with double precision and it fails with an error: TypeError: Input 'filter' of 'Conv2D' Op has type float64 that does not match type float32 of argument 'input'.. The code I am using is as follows:

import numpy as np
from sklearn.utils import shuffle
import keras
from keras.models import Sequential
from keras.layers import Input, Dense, Conv2D, MaxPooling2D, Dropout,Flatten
from keras import backend as K
from keras.models import Model
from keras.utils import np_utils
import time
import tensorflow as tf
K.set_floatx('float64') # Note: the code works if we comment this line, i.e., with single precision
from mlxtend.data import mnist_data
X, y = mnist_data()
X = X.astype(np.float64)
X, y =  shuffle(X, y)
keras_model = Sequential()
keras_model.add(Conv2D(32, kernel_size=(5, 5), activation='relu', input_shape=(28,28,1), padding='same'))
keras_model.add(MaxPooling2D(pool_size=(2, 2)))
keras_model.add(Conv2D(64, (5, 5), activation='relu', padding='same'))
keras_model.add(MaxPooling2D(pool_size=(2, 2)))
keras_model.add(Flatten())
keras_model.add(Dense(512, activation='relu'))
keras_model.add(Dropout(0.5))
keras_model.add(Dense(10, activation='softmax'))
keras_model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.SGD(lr=0.01, momentum=0.95, decay=5e-4, nesterov=True))
keras_model.fit(X.reshape((-1, 28,28, 1)), np_utils.to_categorical(y, 10), epochs=1, batch_size=64)

Any suggestions is much appreciated :)

Answer 1

You have a gaming NVIDIA GPU. You can use only float32 or int32, that's all.
It's the default of TensorFlow.
This default is introduced by Tensorflow due to limitations of CUDA capable GPUs of Nvidia. Best explanation I found here. So the premium Tesla GPUs work well on float16 and float64 as well, but the gaming GPUs work only on float32 and perform very bad for float16 or float64.
I think we are all looking at OpenCL that is supported by AMD GPUs that are more pricey. Unfortunately OpenCL is not supported by TensorFlow as of now.

Suggestion#1: You are stuck with float32. Forget changing it while having that hardware.

Suggestion#2: Once you get a GPU good with float16 change to that. Machine learning doesn't require high precision above that.