How to use F1 Score with Keras model?

Question

For some reason I get error message when trying to specify f1 score with Keras model:

model.compile(optimizer='adam', loss='mse', metrics=['accuracy', 'f1_score'])

I get this error:

ValueError: Unknown metric function:f1_score

After providing 'f1_score' function in the same file where I use 'model.compile' like this:

def f1_score(y_true, y_pred):

    # Count positive samples.
    c1 = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    c2 = K.sum(K.round(K.clip(y_pred, 0, 1)))
    c3 = K.sum(K.round(K.clip(y_true, 0, 1)))

    # If there are no true samples, fix the F1 score at 0.
    if c3 == 0:
        return 0

    # How many selected items are relevant?
    precision = c1 / c2

    # How many relevant items are selected?
    recall = c1 / c3

    # Calculate f1_score
    f1_score = 2 * (precision * recall) / (precision + recall)
    return f1_score 

model.compile(optimizer='adam', loss='mse', metrics=['accuracy', f1_score])

Model compiles all right and can be saved to a file:

model.save(model_path) # works ok

Yet loading it in another program, :

from keras import models 
model = models.load_model(model_path)

fails with an error:

ValueError: Unknown metric function:f1_score

Specifying 'f1_score' in the same file this time does not help, Keras does not see it. What's wrong? How to use F1 Score with Keras model?

Answer 1

When you load the model, you have to supply that metric as part of the custom_objects bag.

Try it like this:

from keras import models 
model = models.load_model(model_path, custom_objects= {'f1_score': f1_score})

Where f1_score is the function that you passed through compile.

Answer 2

For your implementation of f1_score to work I had to switch y_true and y_pred in the function declaration. P.S.: for those who asked: K = keras.backend