How to save Keras model progress into a file?

Question

I have a neural network model coded in Keras.

When I run it on my laptop, I have the following output that shows the progress of the model:

Train on 4 samples, validate on 1 samples Epoch 1/1 4/4 [==============================] - 22s 5s/step - loss: 0.2477 - val_loss: 0.2672

However, when I submit this code to a cluster for running, I do not know how many epochs are left, so I would like to save the above output into a file while the model is running.

How can I do that?

Answer 1

You can do it with a for loop and model.save:

import os

PATH_TO_MODELS = 'path to models directory'

TOTAL_EPOCHS = 8 # number of epochs you want to save

for epoch in range(TOTAL_EPOCHS):
    model.fit(..., epochs = 1)
    save_name = 'model_%sepochs.h5' % str(epoch)
    model.save(os.path.join(PATH_TO_MODELS, save_name))

This will save the output model from each epoch into a distinct file, so that you can keep track of them.

Answer 2

At least one way is to add the remote monitor call back and actually view it in real time. I haven't played with this yet but I know it exists and have wanted to.

keras.callbacks.RemoteMonitor(root='http://localhost:9000', path='/publish/epoch/end/', field='data', headers=None, send_as_json=False)

You can find the documentation here.

Another option is to use TensorBoard. Keras also has a callback for that in the link I already provided. Here is the information from TensorFlow.