Keras: what does class_weight actually try to balance?

Question

My data has extreme class imbalance. About 99.99% of samples are negatives; the positives are (roughly) equally divided among three other classes. I think the models I'm training are just predicting the majority class basically all the time. For this reason, I'm trying to weight the classes.

Model

model = Sequential()

#Layer 1
model.add(Conv1D( {{choice([32, 64, 90, 128])}}, {{choice([3, 4, 5, 6, 8])}}, activation='relu', kernel_initializer=kernel_initializer, input_shape=X_train.shape[1:]))
model.add(BatchNormalization())

#Layer 2
model.add(Conv1D({{choice([32, 64, 90, 128])}}, {{choice([3, 4, 5, 6])}}, activation='relu',kernel_initializer=kernel_initializer))
model.add(Dropout({{uniform(0, 0.9)}}))

#Flatten
model.add(Flatten())

#Output
model.add(Dense(4, activation='softmax'))

(The {{...}} are for use with Hyperas.)

How I've tried to weight it

\1. Using class_weight in model.fit()

model.fit(X_train, Y_train, batch_size=64, epochs=10, verbose=2, validation_data=(X_test, Y_test), class_weight={0: 9999, 1:9999, 2: 9999, 3:1})

\2. Using class_weight in model.fit() with sklearn compute_class_weight()

model.fit(..., class_weight=class_weight.compute_class_weight("balanced", np.unique(Y_train), Y_train)

\3. With a custom loss function

from keras import backend as K
def custom_loss(weights):
    #gist.github.com/wassname/ce364fddfc8a025bfab4348cf5de852d

    def loss(Y_true, Y_pred):
        Y_pred /= K.sum(Y_pred, axis=-1, keepdims=True)
        Y_pred = K.clip(Y_pred, K.epsilon(), 1 - K.epsilon())

        loss = Y_true * K.log(Y_pred) * weights
        loss = -K.sum(loss, -1)
        return loss

    return loss

extreme_weights = np.array([9999, 9999, 9999, 1])
model.compile(loss=custom_loss(extreme_weights),
            metrics=['accuracy'],
            optimizer={{choice(['rmsprop', 'adam', 'sgd','Adagrad','Adadelta'])}}
            )

#(then fit *without* class_weight)

Results

Poor. Accuracy across all classes is ~.99, and unbalanced accuracy for all classes is ~.5. But more meaningful metrics, like auPRC, tell a different story. The auPRC is nearly 1 for the majority class, and nearly 0 for the rest.

Is this how Keras balances classes? It just makes sure that the accuracy is the same across them—or should either metrics be equal or comparable too? Or am I specifying the weights wrong?

Answer 1

Keras uses the class weights during training but the accuracy is not reflective of that. Accuracy is calculated across all samples irrelevant of the weight between classes. This is because you're using the metric 'accuracy' in the compile(). You can define a custom and more accurate weighted accuracy and use that or use the sklearn metrics (e.g. f1_score() which can be 'binary', 'weighted' etc).

Example:

def macro_f1(y_true, y_pred):
     return f1_score(y_true, y_pred, average='macro')


model.compile(loss=custom_loss(extreme_weights),
        metrics=['accuracy', macro_f1],
        optimizer={{choice(['rmsprop', 'adam', 'sgd','Adagrad','Adadelta'])}}
        )