How to use weights to manually predict data in neural network with keras

Question

I've done my best to follow online guides regarding the structure of neural networks, but I must be missing something fundamental. Given a set of trained weights along with their bias, I'd like to simply predict an input manually with those weights without using the predict method.

Using MNIST images with keras I've attempted to manually edit my data to include an extra feature for the bias, however this effort seems to offer no better image accuracy than using no bias at all, and definitely far less accuracy than using the keras predict method. My code is below along with my attempt.

Please note the two comments near the bottom for using the keras method prediction for an accurate image representation, and then my poor attempt from getting the weights manually and adding the bias.

from keras.datasets import mnist
import numpy as np
import time
from keras.models import Sequential
from keras.layers import Dense
import tensorflow as tf
from matplotlib import pyplot as plt

comptime=time.time()
with tf.device('/cpu:0'):
    tf.placeholder(tf.float32, shape=(None, 20, 64))

    seed = 7
    np.random.seed(seed)
    model = Sequential()
    (x_train, _), (x_test, _) = mnist.load_data()
    x_train = x_train.astype('float32') / 255.
    priorShape_x_train=x_train.shape #prior shape of training set
    x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
    x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))
    x_train_shaped=x_train
    model.add(Dense(32, input_dim=784, init='uniform', activation='relu'))
    model.add(Dense(784, init='uniform', activation='sigmoid'))
    model.compile(loss='binary_crossentropy', optimizer='adadelta', metrics=['accuracy'])
    model.fit(x_train[1:2500], x_train[1:2500], nb_epoch=10)

#proper keras prediction
prediction_real=model.predict(x_train[57:58])
prediction_real=prediction_real.reshape((28,28))

#manual weight prediction attempt
x_train=np.hstack([x_train,np.zeros(x_train.shape[0]).reshape(x_train.shape[0],1)]) #add extra column for bias
x_train[:,-1]=1 #add placeholder as 1
weights=np.vstack([model.get_weights()[0],model.get_weights()[1]]) #add trained weights as extra row vector
prediction=np.dot(x_train,weights) #now take dot product.. repeat pattern for next layer
prediction=np.hstack([prediction,np.zeros(prediction.shape[0]).reshape(prediction.shape[0],1)])
prediction[:,-1]=1
weights=np.vstack([model.get_weights()[2],model.get_weights()[3]])
prediction=np.dot(prediction,weights)
prediction=prediction.reshape(priorShape_x_train)

plt.imshow(prediction[57], interpolation='nearest',cmap='gray')
plt.savefig('myprediction.png') #my prediction, not accurate
plt.imshow(prediction_real,interpolation='nearest',cmap='gray')
plt.savefig('realprediction.png') #in-built keras method, accurate

Answer 1

The manual prediction calculation seems to be correct other than the missing activation function like activation='relu' after first layer and activation='sigmoid' in the final layer.

Do the following changes to the manual prediction code and the prediction should work fine:

from scipy.stats import logistic

weights=np.vstack([model.get_weights()[0],model.get_weights()[1]]) 
prediction=np.dot(x_train,weights) 

prediction[prediction<0]=0              ### RELU after 1st layer

prediction=np.hstack([prediction,np.zeros(prediction.shape[0]).reshape(prediction.shape[0],1)])
prediction[:,-1]=1
weights=np.vstack([model.get_weights()[2],model.get_weights()[3]])
prediction=np.dot(prediction,weights)

prediction=logistic.cdf(prediction)     ### Sigmoid after 2nd layer

prediction=prediction.reshape(priorShape_x_train)