CNN gives biased results

Question

I am using a binary classifier on CNN. I have two categories "me" and "others". I have around 250 images of myself and 500 of others ( random faces db ). My current implementation of layers is very simple

    self.model.add(Conv2D(128, (2, 2), padding='same', 
    input_shape=dataset.X_train.shape[1:]))
    self.model.add(Activation('relu'))
    self.model.add(MaxPooling2D(pool_size=(2, 2)))
    self.model.add(Dropout(0.25))

    self.model.add(Conv2D(64, (2, 2), padding='same'))
    self.model.add(Activation('relu'))
    self.model.add(MaxPooling2D(pool_size=(2, 2)))
    self.model.add(Dropout(0.25))

    self.model.add(Conv2D(32, (1, 1), padding='same'))
    self.model.add(Activation('relu'))
    self.model.add(MaxPooling2D(pool_size=(2, 2)))
    self.model.add(Dropout(0.5))
    self.model.add(Dense(512))
    self.model.add(Activation('relu'))
    self.model.add(Dropout(0.25))
    self.model.add(Dense(2)) # for two classes
    self.model.add(Activation('softmax'))

My network reaches accuracy of 93%

My problem is the when I use this network to predict faces it always recognises any face as mine. I have cropped the faces, applied gabor filter but nothing works. Any suggestion will be appreciated.

Prediction Results on Random Faces: [KK represents my face] The probabilities are always over 97%:

KK identified!
1/1 [==============================] - 0s
[[ 0.9741978  0.0258022]]
1/1 [==============================] - 0s

KK identified!
1/1 [==============================] - 0s
[[ 0.9897241   0.01027592]]
1/1 [==============================] - 0s

Prediction Results on my images: [KK represents my face] The probabilities are always over 99%:

KK identified!
1/1 [==============================] - 0s
[[ 0.99639165  0.00360837]]
1/1 [==============================] - 0s
KK identified!
1/1 [==============================] - 0s
[[ 0.99527925  0.00472075]]
1/1 [==============================] - 0s

Training code

   def get_data(self, img_rows=IMAGE_SIZE, img_cols=IMAGE_SIZE, img_channels=3, nb_classes=2):

        images, labels = fetch_data('./data/')
        labels = np.reshape(labels, [-1])

        X_train, X_test, y_train, y_test =  \
            train_test_split(images, labels, test_size=0.3, random_state=random.randint(0, 100))
        X_valid, X_test, y_valid, y_test = \
            train_test_split(images, labels, test_size=0.3, random_state=random.randint(0, 100))
            #train_test_split(images, labels, test_size=0.3, random_state=np.random.seed(15))

        if K.image_dim_ordering() == 'th':
            X_train = X_train.reshape(X_train.shape[0], 3, img_rows, img_cols)
            X_valid = X_valid.reshape(X_valid.shape[0], 3, img_rows, img_cols)
            X_test = X_test.reshape(X_test.shape[0], 3, img_rows, img_cols)
            # input_shape = (3, img_rows, img_cols)
        else:
            X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 3)
            X_valid = X_valid.reshape(X_valid.shape[0], img_rows, img_cols, 3)
            X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 3)
            # input_shape = (img_rows, img_cols, 3)

        Y_train = np_utils.to_categorical(y_train, nb_classes)
        Y_valid = np_utils.to_categorical(y_valid, nb_classes)
        Y_test = np_utils.to_categorical(y_test, nb_classes)

        X_train = X_train.astype('float32')
        X_valid = X_valid.astype('float32')
        X_test = X_test.astype('float32')
        X_train /= 255
        X_valid /= 255
        X_test /= 255

        self.X_train = X_train
        self.X_valid = X_valid
        self.X_test = X_test
        self.Y_train = Y_train
        self.Y_valid = Y_valid
        self.Y_test = Y_test



def train_network(self, dataset, batch_size=32, nb_epoch=40, data_augmentation=True):
    sgd = SGD(lr=0.003, decay=0.0000001, momentum=0.9, nesterov=True)

    # adam = Adam(lr=0.01, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0001)
    self.model.compile(loss='binary_crossentropy',
                       optimizer=sgd,
                       metrics=['accuracy'])
    if not data_augmentation:
        processed_data = self.model.fit(dataset.X_train, dataset.Y_train,
                       batch_size=batch_size,
                       nb_epoch=nb_epoch,
                       validation_data=(dataset.X_valid, dataset.Y_valid),
                       shuffle=True)
    else:
        datagenerator = ImageDataGenerator(
            featurewise_center=False,
            samplewise_center=False,
            featurewise_std_normalization=False,
            samplewise_std_normalization=False,
            zca_whitening=False,
            rotation_range=20,
            width_shift_range=0.2,
            height_shift_range=0.2,
            horizontal_flip=True,
            vertical_flip=False)

        datagenerator.fit(dataset.X_train)

        processed_data = self.model.fit_generator(datagen.flow(dataset.X_train, dataset.Y_train, batch_size=batch_size, shuffle=True),
                                 samples_per_epoch=dataset.X_train.shape[0], nb_epoch=nb_epoch, validation_data=(dataset.X_valid, dataset.Y_valid))

Thanks

[Update: 11th June]

Layers

def build_model(self, dataset, nb_classes=2):
        self.model = Sequential()

        self.model.add(Conv2D(32, (3, 3), padding='same', input_shape=dataset.X_train.shape[1:]))
        self.model.add(Activation('relu'))
        self.model.add(Conv2D(32, (3, 3)))
        self.model.add(Activation('relu'))
        self.model.add(MaxPooling2D(pool_size=(2, 2)))
        self.model.add(Dropout(0.5))

        self.model.add(Conv2D(16, (3, 3), padding='same'))
        self.model.add(Activation('relu'))
        self.model.add(Conv2D(16, (3, 3)))
        self.model.add(Activation('relu'))
        self.model.add(MaxPooling2D(pool_size=(2, 2)))
        self.model.add(Dropout(0.5))

        self.model.add(Flatten())
        self.model.add(Dense(512))
        self.model.add(Activation('relu'))
        self.model.add(Dropout(0.5))
        self.model.add(Dense(nb_classes))
        self.model.add(Activation('softmax'))

        self.model.summary()

Data augmentation

    # this will do preprocessing and realtime data augmentation
    datagen = ImageDataGenerator(
        featurewise_center=True,             # set input mean to 0 over the dataset
        samplewise_center=False,              # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,   # divide each input by its std
        zca_whitening=False,                  # apply ZCA whitening
        rotation_range=20,                     # randomly rotate images in the range (degrees, 0 to 180)
        width_shift_range=0.2,                # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.2,               # randomly shift images vertically (fraction of total height)
        # rescale=1. / 255,
        # shear_range=0.2,
        # zoom_range=0.2,
        horizontal_flip=True,                 # randomly flip images
        vertical_flip=False)                  # randomly flip images

    datagen.fit(dataset.X_train)

    checkpoint = ModelCheckpoint(self.FILE_PATH, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
    callback_list = [checkpoint]

    # fit the model on the batches generated by datagen.flow()
    train_generator = datagen.flow(dataset.X_train, dataset.Y_train, batch_size=batch_size, shuffle=True)

    history = self.model.fit_generator(train_generator,
                                       samples_per_epoch=dataset.X_train.shape[0],
                                       nb_epoch=nb_epoch,
                                       validation_data=(dataset.X_valid, dataset.Y_valid),
                                       callbacks=callback_list)

Dataset

class DataSet(object):

    def __init__(self):
        self.X_train = None
        self.X_valid = None
        self.X_test = None
        self.Y_train = None
        self.Y_valid = None
        self.Y_test = None

    # support only binary classification for now, thus 2 class limit
    def get_data(self, img_rows=IMAGE_SIZE, img_cols=IMAGE_SIZE, img_channels=3, nb_classes=2):

        images, labels = fetch_data('./data/')
        labels = np.reshape(labels, [-1])

        X_train, X_test, y_train, y_test =  \
            train_test_split(images, labels, test_size=0.2, random_state=random.randint(0, 100))

        X_valid, X_test, y_valid, y_test = \
            train_test_split(images, labels, test_size=0.2, random_state=random.randint(0, 100))

        if K.image_dim_ordering() == 'th':
            X_train = X_train.reshape(X_train.shape[0], 3, img_rows, img_cols)
            X_valid = X_valid.reshape(X_valid.shape[0], 3, img_rows, img_cols)
            X_test = X_test.reshape(X_test.shape[0], 3, img_rows, img_cols)

        else:
            X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 3)
            X_valid = X_valid.reshape(X_valid.shape[0], img_rows, img_cols, 3)
            X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 3)


        # convert class vectors to binary class matrices
        Y_train = np_utils.to_categorical(y_train, nb_classes)
        Y_valid = np_utils.to_categorical(y_valid, nb_classes)
        Y_test = np_utils.to_categorical(y_test, nb_classes)

        X_train = X_train.astype('float32')
        X_valid = X_valid.astype('float32')
        X_test = X_test.astype('float32')
        X_train /= 255
        X_valid /= 255
        X_test /= 255

        self.X_train = X_train
        self.X_valid = X_valid
        self.X_test = X_test
        self.Y_train = Y_train
        self.Y_valid = Y_valid
        self.Y_test = Y_test

Answer 1

The result is not odd at all. The network has never learnt what makes your face special, but just remembered what makes the 500 set different from yours. Once you present a new face, it has no "memory" of it, and therefore interprets it as yours, just because none of features present in the 500 faces appears in the 501th.

Some ideas how to tackle this:

• Augment your data with ImageDataGenerator, as proposed by petezurich.
• Increase your kernel size. 2*2 is too tiny to catch facial features. Consider 3*3, or even stacking two 3*3 in the first hidden layer.
• Consider using Batch Normalization and regularization. Increase Dropout to 0.5.
• Consider replacing pooling layers with diluted convolutions (available in Keras).
• Make sure you normalise your input data.
• Lower the number of feature maps (filters) in the first layer. Consider using e.g. 32 3*3 maps instead of 128 tiny elements (these are your main problem if I am to guess). This way you will force the network to generalise instead of learning some tiny nuances.

A nice test for my hypothesis from the last point would be to visualise the activations in hidden layers, especially the first hidden layer. I have a feeling your network activates on some irrelevant features (or rather - noise), rather than "human features" (like eyes, haircut).

[EDIT after more code has been added]

• Centre your input data around zero.
• Lower the batch size. With so few examples you don't want too much of averaging in a batch.

I still think that using e.g. 16 or 32 filters in the first hidden layer should be the first thing to check. Look at your face. Can you spot 128 "features"? Unless you have some severe acne, I don't think so.

Answer 2

For a classification task like that you don´t have enough data with 250 + 500 samples. And a 50/100 relation between class A (you) vs class B (others) is a substantial bias. At least you should try to equalize that with the class_weight parameter in the .fit() function while training.

A better approach would be to retrain an existing ConvNet like VGG16 or Inception from Keras applications: https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

You easily could increase your number of samples with data augmentation (see ImageDataGenerator here https://keras.io/preprocessing/image/).

In order to analyze your code you need to show how you split your data and how you train it. The right way of training is splitting in train data and validation data and than evaluate on another set of separate test data, that the network never has seen and you haven´t optimized your hyperparameters with.

As far as i can see from your comment regarding your train/test/validation split: Do you split two times from the same set of images? That gives you probably the same images in validation and test data, which in turn would lead to wrong results.

And to see your code for training would be helpful.