Test data predictions yield random results when making predictions from a saved model

Question

I am classifying aerial imagery that is tiled into 256x256 tiles using Keras and TensorFlow. The model splits the training data (i.e. the 256x256 image tiles making up the study area) into 70% training data and 30% validation data. A sequential model is used followed by an image data generator. Lastly, a fit generator is used to fit the model to the data. The model is then saved to h5 format to be used to predict classes with other imagery in different study areas.

When I run the model using the 70%/30% training/validation split, the predictions on the validation images work great with increasingly higher accuracies and steadily decreasing loss per epoch. Additionally, when I visualize the predictions (i.e. probability arrays) by joining the probability arrays to vector polygons representing the tile boundaries, the classified results look very good.

My problem is when I use the saved h5 model to make predictions on new imagery--the results are nonsensical and appear random for each tile. It is as if the probability arrays are being shuffled randomly such that when I join the results to the vector image boundary tiles, the results look totally random. How can I resolve this issue?

Here is relevant portions of the code used to train the model:

base_model = applications.VGG16(weights='imagenet', include_top=False, input_shape=(img_rows, img_cols, img_channel))

add_model = Sequential()
add_model.add(Flatten(input_shape=base_model.output_shape[1:]))
add_model.add(Dense(256, activation='relu'))
add_model.add(Dense(n_classes, activation='sigmoid')) # n classes

model = Model(inputs=base_model.input, outputs=add_model(base_model.output))
model.compile(loss='binary_crossentropy', optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])

######################

batch_size = 32
epochs = 50

print('Running the image data generator...')
train_datagen = ImageDataGenerator(
        rotation_range=30, 
        width_shift_range=0.1,
        height_shift_range=0.1, 
        horizontal_flip=True)
train_datagen.fit(x_train)

print('Fitting the model...')
history = model.fit_generator(
    train_datagen.flow(x_train, y_train, batch_size=batch_size),
    steps_per_epoch=x_train.shape[0] // batch_size,
    epochs=epochs,
    #validation_data=(x_valid, y_valid),
    #callbacks=[ModelCheckpoint(model_checkpoint, monitor='val_acc', save_best_only=True)]
)

######################

## Predict
#print('Predicting...')
#p_valid = model.predict(x_valid, batch_size=128)

## Write predictions to csv
#print('Saving predictions to CSV...')
#df = pd.DataFrame(p_valid)
#df['image'] = split + 1 + df.index 
#df.to_csv(out_csv, index=False, header=False)

""" 
Save model, including these details:
-the architecture of the model, allowing to re-create the model
-the weights of the model
-the training configuration (loss, optimizer)
-the state of the optimizer, allowing to resume training exactly where you left off.
"""
print("Saving model")
model.save("/vgg16-model-50epochs.h5")

print('Processing complete.')

---

And the following script uses the saved model from above to make predictions on test imagery from a different study area. Note, there is no 70/30 training/validation split in the final training run above--I simply use 100% of the tiles to train the model, which I then save and reuse in the following script:

import glob, os, time
import cv2
import numpy as np
import pandas as pd

from keras.models import load_model
#from keras.models import model_from_json

# Path to the input tiles which will be used to predict classes
inws = '/image-directory-for-another-study-area'
tiles = glob.glob(os.path.join(inws, '*.tif'))

# h5 file from trained model
in_h5 = "/vgg16-model-50epochs.h5"

# Output model predictions in csv format
out_csv = '/new-predictions.csv'

# Read images and convert to numpy array
x_test = np.array([cv2.imread(tile) for tile in tiles], np.float16) / 255.

print('Loading existing model...')
loaded_model = load_model(in_h5)

print("Predicting on image tiles...")
predictions = loaded_model.predict(x_test, batch_size=128)

# Save to csv
df = pd.DataFrame(predictions)
df['image'] = df.index + 1
df.to_csv(out_csv, index=False, header=False)
print("Predictions saved to disk: {0}".format(out_csv))

Answer 1

I highly suspect this is due to mismatched preprocessing, i.e. you apply different preprocessing for x_train and x_test.

Since you didn't show how x_train is obtained, so I can't verify for you. However, it is known that the pretrained VGG16 model uses caffe-like normalization (see preprocess_input), which normalizes an input image by subtracting channel-wise mean. Note, this is different from what you did for testing images,

x_test = np.array([cv2.imread(tile) for tile in tiles], np.float16) / 255.

Instead, you need to do something in the keras VGG16 example

#Extract features with VGG16
from keras.applications.vgg16 import VGG16
from keras.preprocessing import image
from keras.applications.vgg16 import preprocess_input
import numpy as np

model = VGG16(weights='imagenet', include_top=False)

img_path = 'elephant.jpg'
img = image.load_img(img_path, target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x) <---- use this function to preprocess your image

features = model.predict(x)

Answer 2

In the second script, the use of glob creates a list of tiff files that are unordered. For this approach to work, you need an ordered list of tiff files (e.g. [00001.tif, 00002.tif, ... 1234.tif]) that can be associated with the ordered predictions. The sorted() function can be used to do the ordering.

tiles = sorted(glob.glob(os.path.join(inws, '*.tif')))