Got different accuracy for training model and load model

Question

I've tried both model.save_weights() and model.save(), and their corresponding load statements (model.load_weights() and load_model()). If I just push all my evaluation code at the end of the training code, things work out fine.

The problem is with stopping Python, starting a new Python script, and reading in the weights/model to use them for inference. The main method I've tried when loading to: define the model (using same code from training run that saved the weights), then run model.load_weights(), then compile the model. This is what's not working. And yet, as I say, going the model.save() and load_model() route produces similar garbage output.

My Code:

from tensorflow import keras
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Flatten, Dense, Embedding, Conv1D, GlobalMaxPooling1D, Dropout
from tensorflow.keras.callbacks import EarlyStopping, ReduceLROnPlateau, ModelCheckpoint

model = Sequential()
model.add(Embedding(vocab_size, embedding_dim, input_length=train_padded.shape[1]))

model.add(Conv1D(48, 5, activation='relu', padding='valid'))
model.add(GlobalMaxPooling1D())
model.add(Dropout(0.5))

model.add(Flatten())
model.add(Dropout(0.5))

model.add(Dense(11, activation='softmax'))
model.compile(loss= 'categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

epochs = 50
batch_size = 32
history = model.fit(train_padded, training_labels, shuffle=False ,
                    epochs=epochs, batch_size=batch_size,
                    validation_split=0.2,
                    callbacks=[ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.0001),
                               EarlyStopping(monitor='val_loss', mode='min', patience=2, verbose=1),
                               EarlyStopping(monitor='val_accuracy', mode='max', patience=5, verbose=1)])
score = model.evaluate(train_padded, training_labels, verbose=0)
print("%s: %.2f%%" % (model.metrics_names[1], score[1]*100))

#save model
tf.keras.models.save_model(model,'model.h5',overwrite=True)
#load model
model = tf.keras.models.load_model('model.h5')
score = model.evaluate(train_padded, training_labels, verbose=0)
print("%s: %.2f%%" % (model.metrics_names[1], score[1]*100))

#or
#save weights
model.save_weights('model.h5')
#load weights
model.load_weights('model.h5')
model.compile(loss= 'categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
score = model.evaluate(train_padded, training_labels, verbose=0)
    print("%s: %.2f%%" % (model.metrics_names[1], score[1]*100))

Output

83.14% #training model
17.60% #load_model
21.37% #load_weights

Please help me.. The weights for training model and load model is same but the accuracy is different. The accuracy when using load_model() has gone from ~80% (training model) to ~20% on the same data.

Thanks

Answer 1

My issue got solved by fixing the seed for keras which uses NumPy random generator and since I am using Tensorflow as backend, I also fixed the seed for it. These 4 lines I added at the top of my file where the model is also defined.

from numpy.random import seed 
seed(1) # keras seed fixing import

import tensorflow
tensorflow.random.set_seed(2) # tensorflow seed fixing

For more information have a look at this- https://machinelearningmastery.com/reproducible-results-neural-networks-keras/