What does initial_epoch in Keras mean?

Question

I'm a little bit confused about initial_epoch value in fit and fit_generator methods. Here is the doc:

initial_epoch: Integer. Epoch at which to start training (useful for resuming a previous training run).

I understand, it is not useful if you start training from scratch. It is useful if you trained your dataset and want to improve accuracy or other values (correct me if I'm wrong). But I'm not sure what it really does.

So after all this, I have 2 questions:

1. What does initial_epoch do and what is it for?
2. When can I use initial_epoch?

• When I change my dataset?
• When I change the learning rate, optimizer or loss function?
• Both of them?

Answer 1

Since in some of the optimizers, some of their internal values (e.g. learning rate) are set using the current epoch value, or even you may have (custom) callbacks that depend on the current value of epoch, the initial_epoch argument let you specify the initial value of epoch to start from when training.

As stated in the documentation, this is mostly useful when you have trained your model for some epochs, say 10, and then saved it and now you want to load it and resume the training for another 10 epochs without disrupting the state of epoch-dependent objects (e.g. optimizer). So you would set initial_epoch=10 (i.e. we have trained the model for 10 epochs) and epochs=20 (not 10, since the total number of epochs to reach is 20) and then everything resume as if you were initially trained the model for 20 epochs in one single training session.

However, note that when using built-in optimizers of Keras you don't need to use initial_epoch, since they store and update their state internally (without considering the value of current epoch) and also when saving a model the state of the optimizer will be stored as well.