Imagine a fully-connected neural network with its last two layers of the following structure:
[Dense] units = 612 activation = softplus [Dense] units = 1 activation = sigmoid
The output value of the net is 1, but I'd like to know what the input x to the sigmoidal function was (must be some high number, since sigm(x) is 1 here).
Folllowing indraforyou's answer I managed to retrieve the output and weights of Keras layers:
outputs = [layer.output for layer in model.layers[-2:]] functors = [K.function( [model.input]+[K.learning_phase()], [out] ) for out in outputs] test_input = np.array(...) layer_outs = [func([test_input, 0.]) for func in functors] print layer_outs[-1][0] # -> array([[ 1.]]) dense_0_out = layer_outs[-2][0] # shape (612, 1) dense_1_weights = model.layers[-1].weights[0].get_value() # shape (1, 612) dense_1_bias = model.layers[-1].weights[1].get_value() x = np.dot(dense_0_out, dense_1_weights) + dense_1_bias print x # -> -11.7
How can x be a negative number? In that case the last layers output should be a number closer to 0.0 than 1.0. Are dense_0_out
or dense_1_weights
the wrong outputs or weights?
Since you're using get_value()
, I'll assume that you're using Theano backend. To get the value of the node before the sigmoid activation, you can traverse the computation graph.
The graph can be traversed starting from outputs (the result of some computation) down to its inputs using the owner field.
In your case, what you want is the input x
of the sigmoid activation op. The output of the sigmoid op is model.output
. Putting these together, the variable x
is model.output.owner.inputs[0]
.
If you print out this value, you'll see Elemwise{add,no_inplace}.0
, which is an element-wise addition op. It can be verified from the source code of Dense.call()
:
def call(self, inputs): output = K.dot(inputs, self.kernel) if self.use_bias: output = K.bias_add(output, self.bias) if self.activation is not None: output = self.activation(output) return output
The input to the activation function is the output of K.bias_add()
.
With a small modification of your code, you can get the value of the node before activation:
x = model.output.owner.inputs[0] func = K.function([model.input] + [K.learning_phase()], [x]) print func([test_input, 0.])
For anyone using TensorFlow backend: use x = model.output.op.inputs[0]
instead.
I can see a simple way just changing a little the model structure. (See at the end how to use the existing model and change only the ending).
The advantages of this method are:
There are two possible solutions below:
Model structure
You could just have the last dense separated in two layers at the end:
[Dense] units = 612 activation = softplus [Dense] units = 1 #no activation [Activation] activation = sigmoid
Then you simply get the output of the last dense layer.
I'd say you should create two models, one for training, the other for checking this value.
Option 1 - Building the models from the beginning:
from keras.models import Model #build the initial part of the model the same way you would #add the Dense layer without an activation: #if using the functional Model API denseOut = Dense(1)(outputFromThePreviousLayer) sigmoidOut = Activation('sigmoid')(denseOut) #if using the sequential model - will need the functional API model.add(Dense(1)) sigmoidOut = Activation('sigmoid')(model.output)
Create two models from that, one for training, one for checking the output of dense:
#if using the functional API checkingModel = Model(yourInputs, denseOut) #if using the sequential model: checkingModel = model trainingModel = Model(checkingModel.inputs, sigmoidOut)
Use trianingModel
for training normally. The two models share weights, so training one is training the other.
Use checkingModel
just to see the outputs of the Dense layer, using checkingModel.predict(X)
Option 2 - Building this from an existing model:
from keras.models import Model #find the softplus dense layer and get its output: softplusOut = oldModel.layers[indexForSoftplusLayer].output #or should this be the output from the dropout? Whichever comes immediately after the last Dense(1) #recreate the dense layer outDense = Dense(1, name='newDense', ...)(softPlusOut) #create the new model checkingModel = Model(oldModel.inputs,outDense)
It's important, since you created a new Dense layer, to get the weights from the old one:
wgts = oldModel.layers[indexForDense].get_weights() checkingModel.get_layer('newDense').set_weights(wgts)
In this case, training the old model will not update the last dense layer in the new model, so, let's create a trainingModel:
outSigmoid = Activation('sigmoid')(checkingModel.output) trainingModel = Model(checkingModel.inputs,outSigmoid)
Use checkingModel
for checking the values you want with checkingModel.predict(X)
. And train the trainingModel
.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With