When using mectrics in model.compile in keras, report ValueError: ('Unknown metric function', ':f1score')

Question

I'm trying to run a LSTM, and when I use the code below:

model.compile(optimizer='rmsprop', loss='binary_crossentropy',
              metrics=['accuracy', 'f1score', 'precision', 'recall'])

It returns:

ValueError: ('Unknown metric function', ':f1score').

I've done my searches and found this url: https://github.com/fchollet/keras/issues/5400

The "metrics" in the "model.compile" part in this url is exactly the same as mine, and no errors are returned.

Answer 1

I suspect you are using Keras 2.X. As explained in https://keras.io/metrics/, you can create custom metrics. These metrics appear to take only (y_true, y_pred) as function arguments, so a generalized implementation of fbeta is not possible.

Here is an implementation of f1_score based on the keras 1.2.2 source code.

import keras.backend as K

def f1_score(y_true, y_pred):

    # Count positive samples.
    c1 = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
    c2 = K.sum(K.round(K.clip(y_pred, 0, 1)))
    c3 = K.sum(K.round(K.clip(y_true, 0, 1)))

    # If there are no true samples, fix the F1 score at 0.
    if c3 == 0:
        return 0

    # How many selected items are relevant?
    precision = c1 / c2

    # How many relevant items are selected?
    recall = c1 / c3

    # Calculate f1_score
    f1_score = 2 * (precision * recall) / (precision + recall)
    return f1_score

To use, simply add f1_score to your list of metrics when you compile your model, after defining the custom metric. For example:

model.compile(loss='categorical_crossentropy',
              optimizer='adam', 
              metrics=['accuracy',f1_score])