Menu
I am developing a model using nadam optimizer. I was wondering if there is a way to switch to sgd during training if validation loss does not reduce for two epochs.
941
The constant learning rate is the default schedule in all Keras Optimizers. For example, in the SGD optimizer, the learning rate defaults to 0.01 . To use a custom learning rate, simply instantiate an SGD optimizer and pass the argument learning_rate=0.01 .
Optimizers are Classes or methods used to change the attributes of your machine/deep learning model such as weights and learning rate in order to reduce the losses. Optimizers help to get results faster.
A typical way is to to drop the learning rate by half every 10 epochs. To implement this in Keras, we can define a step decay function and use LearningRateScheduler callback to take the step decay function as argument and return the updated learning rates for use in SGD optimizer.
Creating a custom optimizer If you intend to create your own optimization algorithm, simply inherit from this class and override the following methods: _resource_apply_dense (update variable given gradient tensor is a dense tf. Tensor ) _resource_apply_sparse (update variable given gradient tensor is a sparse tf.
You can use a learning rate schedule to modulate how the learning rate of your optimizer changes over time: Check out the learning rate schedule API documentation for a list of available schedules. These methods and attributes are common to all Keras optimizers. Apply gradients to variables. This is the second part of minimize ().
This object is used later for training and testing the neural network. Before the model can be trained, Keras requires us to specify some details about the training process like the optimizer, and a loss function. For the example, we also tell Keras to track the network’s accuracy during the training process.
You can use a learning rate schedule to modulate how the learning rate of your optimizer changes over time: Check out the learning rate schedule API documentation for a list of available schedules. These methods and attributes are common to all Keras optimizers.
If you really need to have the loss attribute of your model changed, you can set the compiled_loss attribute using a keras.engine.compile_utils.LossesContainer ( here is the reference) and set model.train_function to model.make_train_function () (so that the new loss gets taken into account).
Would something like this work ?
model.compile( optimizer='Adam', ...)
model.fit( X, y, epochs=100, callback=[EarlyStoppingCallback] )
# now switch to SGD and finish training
model.compile( optimizer='SGD', ...)
model.fit( X, y, epochs=10 )
Or would the second call to compile over-write all the variables (ie. do something like tf.initialize_all_variables()
(It's actually a followup question - but I'm writing this as an answer - because stackoverflow does not allow code in comments)
197
You can create an EarlyStopping callback that will stop the training, and in this callback, you create a function to change your optimizer and fit again.
The following callback will monitor the validation loss (val_loss) and stop training after two epochs (patience) without an improvement greater than min_delta.
min_delta = 0.000000000001
stopper = EarlyStopping(monitor='val_loss',min_delta=min_delta,patience=2)
But for adding an extra action after the training is finished, we can extend this callback and change the on_train_end method:
class OptimizerChanger(EarlyStopping):
def __init__(self, on_train_end, **kwargs):
self.do_on_train_end = on_train_end
super(OptimizerChanger,self).__init__(**kwargs)
def on_train_end(self, logs=None):
super(OptimizerChanger,self).on_train_end(self,logs)
self.do_on_train_end()
For the custom function to call when the model ends training:
def do_after_training():
#warining, this creates a new optimizer and,
#at the beginning, it might give you a worse training performance than before
model.compile(optimizer = 'SGD', loss=...., metrics = ...)
model.fit(.....)
Now let's use the callbacks:
changer = OptimizerChanger(on_train_end= do_after_training,
monitor='val_loss',
min_delta=min_delta,
patience=2)
model.fit(..., ..., callbacks = [changer])
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
