Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can't change activations in existing Keras model

I have a normal VGG16 model with relu activations, i.e.

def VGG_16(weights_path=None):
    model = Sequential()
    model.add(ZeroPadding2D((1, 1),input_shape=(3, 224, 224)))
    model.add(Convolution2D(64, 3, 3, activation='relu'))
    model.add(ZeroPadding2D((1, 1)))
    model.add(Convolution2D(64, 3, 3, activation='relu'))
    model.add(MaxPooling2D((2, 2), strides=(2, 2)))
[...]
    model.add(Flatten())
    model.add(Dense(4096, activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(4096, activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(1000, activation='softmax'))

    if weights_path:
        model.load_weights(weights_path)

    return model

and I'm instantiating it with existing weights and now want to change all relu activations to softmax (not useful, I know)

model = VGG_16('vgg16_weights.h5')
sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)

softmax_act = keras.activations.softmax
for (n, layer) in enumerate(model.layers):
    if 'activation' in layer.get_config() and layer.get_config()['activation'] == 'relu':
        print('replacing #{}: {}, {}'.format(n, layer, layer.activation))
        layer.activation = softmax_act
        print('-> {}'.format(layer.activation))

model.compile(optimizer=sgd, loss='categorical_crossentropy')

Note: model.compile is called after the changes, so the model should still be modifiable I guess.

However, even though the debug-prints correctly say

replacing #1: <keras.layers.convolutional.Convolution2D object at 0x7f7d7c497f50>, <function relu at 0x7f7dbe699a28>
-> <function softmax at 0x7f7d7c4972d0>
[...]

the actual results are identical to the model with relu activations.
Why doesn't Keras use the changed activation function?

like image 860
RickOrMorty Avatar asked Mar 26 '17 15:03

RickOrMorty


People also ask

What is activation =' ReLU in keras?

relu function Applies the rectified linear unit activation function. With default values, this returns the standard ReLU activation: max(x, 0) , the element-wise maximum of 0 and the input tensor.

How do you make a keras layer not trainable?

Freeze the required layers In Keras, each layer has a parameter called “trainable”. For freezing the weights of a particular layer, we should set this parameter to False, indicating that this layer should not be trained.

What is the default activation layer in keras?

Activation. This parameter sets the element-wise activation function to be used in the dense layer. By default, we can see that it is set to None. That means that by default it is a linear activation.


Video Answer


2 Answers

The function utils.apply_modifications() did not work for me. It gave me a warning

WARNING:tensorflow:No training configuration found in save file: the model was not compiled. Compile it manually.

I then recompiled the model then it worked. For illustration, I changed all activation to sigmoid. see the example below

from tensorflow.keras.activations import relu,sigmoid,elu
from tensorflow.keras.applications.vgg16 import VGG16
base_model = VGG16(weights='imagenet', include_top=False,pooling='avg',input_shape= 
    (100, 100, 3))
# before if you check 
base_model.get_config() # you will see all activation are relu 
for layer in base_model.layers:
    if (hasattr(layer,'activation'))==True:
         layer.activation = sigmoid
# without compiling you should not see any changes
# when calling base_model.get_config()
# when compiling
base_model.compile(loss="categorical_crossentropy") #it forced me to put the loss
# now you will see the changes when calling
base_model.get_config()
like image 59
burhan rashid Avatar answered Sep 18 '22 09:09

burhan rashid


you might want to use apply_modifications

idx_of_layer_to_change = -1
model.layers[idx_of_layer_to_change].activation = activations.softmax
model = utils.apply_modifications(model)
like image 39
ahmedhosny Avatar answered Sep 21 '22 09:09

ahmedhosny