is it possible to implement dynamic class weights in keras?

Question

keras supports the class_weights feature to allow giving different classes different weights - for example for when the number of samples is imbalanced

I want to do something similar, but to use dynamic weights, based on the class imbalance in each batch.

is this possible?

Answer 1

Option 1:

Make a manual loop for epochs and batches, use the method train_on_batch, which also accepts class_weight:

for epoch in range(epochs):
    for batchX,batchY in batches: #adapt this loop to your way of creating/getting batches

        weights = calculateOrGetTheWeights(batch)
        model.train_on_batch(batchX,batchY,...,class_weight=weights)

Option 2:

Create a custom loss. May be more tricky and depends on the data format, the number of classes, the type of loss function, etc.

Assuming 2D data (samples, classes) and a multiclass problem:

import keras.backend as K

def customLoss(yTrue,yPred):

    classes = K.argmax(yTrue)
    classCount = K.sum(yTrue,axis=0)

    loss = K.some_loss_function(yTrue,yPred)

    return loss / K.gather(classCount, classes)

Assuming a binary classification (1 class only) with 1D or 2D data:

import keras.backend as K

def binaryCustomLoss(yTrue,yPred):

    positives = yTrue
    negatives = 1 - yTrue

    positiveRatio = K.mean(positives)
    negativeRatio = 1 - positiveRatio #or K.mean(negatives)

    weights = (positives / positiveRatio) + (negatives / negativeRatio)

    #you may need K.squeeze(weights) here

    return weights * K.some_loss_function(yTrue,yPred)

Warning: both loss functions will return Nan (or infinity) if any class count is zero.

Answer 2

One option is to, instead of using class_weight, use samples weights

If you want your sample weight to be dynamic you'll need to use fit_generator instead of fit, so you can change the weights on the run

So in pseudo code:

def gen(x, y):
    while True:
        for x_batch, y_batch in make_batches(x, y):
            weights = make_weights(y_batch)
            yield x_batch, y_batch, weights
model.fit_generator(gen(x_train, y_train))

In this code, make_weights should return an array with the same length as y_batch. Each element is a weight to be applied to the respective sample

If you're unsure about the behavior of class_weight and sample weights being the same, notice how keras standardizes class weights.

So class weights are actually translated to sample weights in the end :)