how to implement custom metric in keras?

Question

I get this error :

sum() got an unexpected keyword argument 'out'

when I run this code:

import pandas as pd, numpy as np import keras from keras.layers.core import Dense, Activation from keras.models import Sequential  def AUC(y_true,y_pred):     not_y_pred=np.logical_not(y_pred)     y_int1=y_true*y_pred     y_int0=np.logical_not(y_true)*not_y_pred     TP=np.sum(y_pred*y_int1)     FP=np.sum(y_pred)-TP     TN=np.sum(not_y_pred*y_int0)     FN=np.sum(not_y_pred)-TN     TPR=np.float(TP)/(TP+FN)     FPR=np.float(FP)/(FP+TN)     return((1+TPR-FPR)/2)  # Input datasets  train_df = pd.DataFrame(np.random.rand(91,1000)) train_df.iloc[:,-2]=(train_df.iloc[:,-2]>0.8)*1   model = Sequential() model.add(Dense(output_dim=60, input_dim=91, init="glorot_uniform")) model.add(Activation("sigmoid")) model.add(Dense(output_dim=1, input_dim=60, init="glorot_uniform")) model.add(Activation("sigmoid"))  model.compile(optimizer='rmsprop',loss='binary_crossentropy',metrics=[AUC])   train_df.iloc[:,-1]=np.ones(train_df.shape[0]) #bias X=train_df.iloc[:,:-1].values Y=train_df.iloc[:,-1].values print X.shape,Y.shape  model.fit(X, Y, batch_size=50,show_accuracy = False, verbose = 1) 

Is it possible to implement a custom metric aside from doing a loop on batches and editing the source code?

Answer 1

Here I'm answering to OP's topic question rather than his exact problem. I'm doing this as the question shows up in the top when I google the topic problem.

You can implement a custom metric in two ways.

1. As mentioned in Keras docu.

   import keras.backend as K  def mean_pred(y_true, y_pred):     return K.mean(y_pred)  model.compile(optimizer='sgd',           loss='binary_crossentropy',           metrics=['accuracy', mean_pred]) 

   But here you have to remember as mentioned in Marcin Możejko's answer that y_true and y_pred are tensors. So in order to correctly calculate the metric you need to use keras.backend functionality. Please look at this SO question for details How to calculate F1 Macro in Keras?

2. Or you can implement it in a hacky way as mentioned in Keras GH issue. For that you need to use callbacks argument of model.fit.

   import keras as keras import numpy as np from keras.optimizers import SGD from sklearn.metrics import roc_auc_score  model = keras.models.Sequential() # ... sgd = SGD(lr=0.001, momentum=0.9) model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy'])   class Metrics(keras.callbacks.Callback):     def on_train_begin(self, logs={}):         self._data = []      def on_epoch_end(self, batch, logs={}):         X_val, y_val = self.validation_data[0], self.validation_data[1]         y_predict = np.asarray(model.predict(X_val))          y_val = np.argmax(y_val, axis=1)         y_predict = np.argmax(y_predict, axis=1)          self._data.append({             'val_rocauc': roc_auc_score(y_val, y_predict),         })         return      def get_data(self):         return self._data  metrics = Metrics() history = model.fit(X_train, y_train, epochs=100, validation_data=(X_val, y_val), callbacks=[metrics]) metrics.get_data()