Is it possible to continue training from a specific epoch?

Question

A resource manager I'm using to fit a Keras model limits the access to a server to 1 day at a time. After this day, I need to start a new job. Is it possible with Keras to save the current model at epoch K, and then load that model to continue training epoch K+1 (i.e., with a new job)?

Answer 1

You can save weights after every epoch by specifying a callback:

weight_save_callback = ModelCheckpoint('/path/to/weights.{epoch:02d}-{val_loss:.2f}.hdf5', monitor='val_loss', verbose=0, save_best_only=False, mode='auto')
model.fit(X_train,y_train,batch_size=batch_size,nb_epoch=nb_epoch,callbacks=[weight_save_callback])

This will save the weights after every epoch. You can then load them with:

model = Sequential()
model.add(...)
model.load('path/to/weights.hf5')

Of course your model needs to be the same in both cases.

Answer 2

You can add the initial_epoch argument. This will allow you to continue training from a specific epoch.