Does interrupting keras training in a Jupyter notebook save the training?

Question

So, I was working on a machine learning project using a Jupyter Notebook and Keras, and I started training. I came back a few hours later, only to realize that I had accidentally set the epochs to a really high number.

I'm wondering, if I stop running the cell (send a KeyboardInterrupt), will the whole training be canceled? Or will the weights from the epoch I was currently on still be saved?

I can still access the model from the next cell.

Answer 1

The trained model will still be in memory, in the state it was in when the KeyboardInterrupt happened. As long as the Python kernel isn't stopped or the model isn't reinstantiated, you can continue to use the trained model. To test this, evaluate the model's prediction accuracy.

Note that, if you continue training the model, a KeyboardInterrupt restarts the epoch counter. That will effect any callbacks that rely on the epoch number.

Answer 2

If you have not defined a ModelCheckpoint callback or some custom model saver callback then the answer is no.

Next time you should include the ModelCheckpoint callback, so at every epoch your model will be saved, and you can restore it