Keras save_weights and ModelCheckpoint Difference

Question

I save keras model by two ways 1. "ModelCheckpoint" 2. "save_weights" after training model

But performance of those two are different when load trained model using "load_weights" and "predict"

My code is as follows

Train & Save Model

model_checkpoint = ModelCheckpoint("Model_weights.hdf5", verbose=1, save_best_only=True)

early_stopping = EarlyStopping(monitor='val_loss', patience=20, verbose=1, restore_best_weights=True)

hist = Model.fit(x=train_dict, y=train_label, 
                       batch_size=batch_size, epochs=epochs, 
                       validation_data=(valid_dict, valid_label),
                       callbacks=[csv_logger, early_stopping, model_checkpoint])

Model.save_weights("Model_weights.h5")

Load Trained Model and Test

Model = create_model() # Construct model skeleton 

hdf5_model = load_model("Model_weights.hdf5")

h5_model = load_model("Model_weights.h5")

There are difference between "hdf5_model.predict(train)" and "h5_model.predict(train)"

Answer 1

First, you need to understand what ModelCheckpoint actually does. It saves only the best weight. You can see the loss and accuracy for each epoch during training. It changes on each epoch. Sometimes it increases and sometimes it decreases as the model continuously updating its weights.

Let's assume a situation. You're training your model for 50 epochs. It's possible that you will get loss = 0.25 on the 45th epoch and loss = 0.37 on the 50th epoch. It's very normal. ModelCheckpoint will only save 45th epochs weight. It won't update on the 50th epoch. ModelCheckpoint only saves the weight only if loss decreases(you can also change the logic via parameter). But if you save the weights after training is completed, it will save with a loss of 0.37 which is higher.

It's very normal that the model saved via ModelCheckpoint has lower loss value and final model has a higher value. That's why you're getting different predictions from these two models.

If you take a look at the graph below, you can see the best loss value was achieved on the 98th epoch. So your ModelCheckpoint saving the weights on 98th epoch and never updating it.