How to get current learning rate of SGD optimizer in TensorFlow 2.0 when I use tf.keras.optimizers.schedules.ExponentialDecay?

Question

I want to reduce learning rate in SGD optimizer of tensorflow2.0, I used this line of code:

lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
            initial_learning_rate=self.parameter['learning_rate'],
            decay_steps=(1000), 
            decay_rate=self.parameter['lr_decay']
        )
opt = tf.keras.optimizers.SGD(learning_rate=lr_schedule, momentum=0.9)

But I don't know if my learning rate has dropped, how can I get my current learning rate?

Answer 1

_decayed_lr method decays the learning_rate based on the number of iterations as and returns the actual learning rate at that specific iteration. It also casts the returned value to a type that you specify. So, the following code can do the job for you:

opt._decayed_lr(tf.float32)

Answer 2

@Lisanu's answer worked for me as well.
Here's why&how that answer works:

This tensorflow's github webpage shows the codes for tf.keras.optimizers.
If you scroll down, there is a function named _decayed_lr which allows users to get the decayed learning rate as a Tensor with dtype=var_dtype.

Therefore, by using optimizer._decayed_lr(tf.float32), we can get the current decayed learning rate.

If you'd like to print the current decayed learning rate during training in Tensorflow, you can define a custom-callback class and utilize optimizer._decayed_lr(tf.float32). The example is as follows:

class CustomCallback(tf.keras.callbacks.Callback):
    def on_epoch_begin(self, epoch, logs=None):
        current_decayed_lr = self.model.optimizer._decayed_lr(tf.float32).numpy()
        print("current decayed lr: {:0.7f}".format(current_decayed_lr))