Menu
Is there a way to set a global weight decay in Keras?
I know about the layer wise one using regularizers(https://keras.io/regularizers/), but I could not find any information about a way to set a global weight decay.
223
To get global weight decay in keras regularizers have to be added to every layer in the model. In my models these layers are batch normalization (beta/gamma regularizer) and dense/convolutions (W_regularizer/b_regularizer) layers. Layer wise regularization is described here: (https://keras.io/regularizers/).
Weight Decay, or Regularization, is a regularization technique applied to the weights of a neural network. We minimize a loss function compromising both the primary loss function and a penalty on the Norm of the weights: L n e w ( w ) = L o r i g i n a l ( w ) + λ w T w.
Weight decay is also known as L2 regularization, because it penalizes weights according to their L2 norm. In weight decay technique, the objective function of minimizing the prediction loss on the training data is replaced with the new objective function, minimizing the sum of the prediction loss and the penalty term.
In the tests we ran, the best learning rate with L2 regularization was 1e-6 (with a maximum learning rate of 1e-3) while 0.3 was the best value for weight decay (with a learning rate of 3e-3).
There is no way to directly apply a "global" weight decay to a whole keras model at once.
However, as I describe here, you can employ weight decay on a model by looping through its layers and manually applying the regularizers on appropriate layers. Here's the relevant code snippet:
model = keras.applications.ResNet50(include_top=True, weights='imagenet')
alpha = 0.00002 # weight decay coefficient
for layer in model.layers:
if isinstance(layer, keras.layers.Conv2D) or isinstance(layer, keras.layers.Dense):
layer.add_loss(lambda layer=layer: keras.regularizers.l2(alpha)(layer.kernel))
if hasattr(layer, 'bias_regularizer') and layer.use_bias:
layer.add_loss(lambda layer=layer: keras.regularizers.l2(alpha)(layer.bias))
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
