Custom TensorFlow metric: true positive rate at given false positive rate

Question

I have a binary classification problem with categories background (bg) = 0, signal (sig) = 1, for which I am training NNs. For monitoring purposes, I am trying to implement a custom metric in Keras with TensorFlow backend that does the following:

1) Calculate the threshold on my NN output which would result in a false positive rate (classifying bg as signal) of X (in this case X = 0.02, but it could be anything).

2) Calculate the true positive rate at this threshold.

Given numpy arrays y_true, y_pred, I would write a function like:

def eff_at_2percent_metric(y_true, y_pred):
    #Find list of bg events
    bg_list = np.argwhere(y_true < 0.5)
    #Order by the NN output
    ordered_bg_predictions = np.flip(np.sort(y_pred[bg_list]),axis=0)
    #Find the threshold with 2% false positive rate
    threshold = ordered_bg_predictions[0.02*round(len(ordered_bg_list))]

    #Find list of signal events
    sig_list = np.argwhere(y_true > 0.5)
    #Order these by NN output
    ordered_sig_predictions = np.sort(y_pred[sig_list])
    #Find true positive rate with this threshold
    sig_eff = 1 - np.searchsorted(ordered_sig_predictions,threshold)/len(ordered_sig_predictions)

    return sig_eff

Of course, this does not work because to implement a custom metric, y_true and y_pred are supposed to be TensorFlow tensors rather than numpy arrays. Is there any way I can make this work correctly?

Answer 1

There's a metric for sensitivity at specificity, which I believe is equivalent (specificity is one minus FPR).