Using pure numpy metric as metric in Keras/TensorFlow

Question

I am working on a competition on Kaggle, where the evaluation metric is defined as

This competition is evaluated on the mean average precision at different intersection over union (IoU) thresholds. The IoU of a proposed set of object pixels and a set of true object pixels is calculated as:

              IoU(A,B)=(A∩B)/(A∪B)

The metric sweeps over a range of IoU thresholds, at each point calculating an average precision value. The threshold values range from 0.5 to 0.95 with a step size of 0.05: (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95). In other words, at a threshold of 0.5, a predicted object is considered a "hit" if its intersection over union with a ground truth object is greater than 0.5. At each threshold value t, a precision value is calculated based on the number of true positives (TP), false negatives (FN), and false positives (FP) resulting from comparing the predicted object to all ground truth objects:

                     TP(t)/TP(t)+FP(t)+FN(t).

A true positive is counted when a single predicted object matches a ground truth object with an IoU above the threshold. A false positive indicates a predicted object had no associated ground truth object. A false negative indicates a ground truth object had no associated predicted object. The average precision of a single image is then calculated as the mean of the above precision values at each IoU threshold:

           (1/|thresholds|)*∑tTP(t)/TP(t)+FP(t)+FN(t)

Now, I have written this function in pure numpy as it was much easier to code in that and I have decorated it with tf.py_fucn() in order to use with it with Keras. Here is the sample code:

def iou_metric(y_true_in, y_pred_in, fix_zero=False):
    labels = y_true_in
    y_pred = y_pred_in

    true_objects = 2
    pred_objects = 2

    if fix_zero:
        if np.sum(y_true_in) == 0:
            return 1 if np.sum(y_pred_in) == 0 else 0

    intersection = np.histogram2d(labels.flatten(), y_pred.flatten(), bins=(true_objects, pred_objects))[0]

    # Compute areas (needed for finding the union between all objects)
    area_true = np.histogram(labels, bins = true_objects)[0]
    area_pred = np.histogram(y_pred, bins = pred_objects)[0]
    area_true = np.expand_dims(area_true, -1)
    area_pred = np.expand_dims(area_pred, 0)

    # Compute union
    union = area_true + area_pred - intersection

    # Exclude background from the analysis
    intersection = intersection[1:,1:]
    union = union[1:,1:]
    union[union == 0] = 1e-9

    # Compute the intersection over union
    iou = intersection / union

    # Precision helper function
    def precision_at(threshold, iou):
        matches = iou > threshold
        true_positives = np.sum(matches, axis=1) == 1   # Correct objects
        false_positives = np.sum(matches, axis=0) == 0  # Missed objects
        false_negatives = np.sum(matches, axis=1) == 0  # Extra objects
        tp, fp, fn = np.sum(true_positives), np.sum(false_positives), np.sum(false_negatives)
        return tp, fp, fn

    # Loop over IoU thresholds
    prec = []

    for t in np.arange(0.5, 1.0, 0.05):
        tp, fp, fn = precision_at(t, iou)
        if (tp + fp + fn) > 0:
            p = tp / (tp + fp + fn)
        else:
            p = 0
        prec.append(p)

    return np.mean(prec)

I tried to convert it into pure tf function but was unable to do it as I am not able to figure out how the control dependencies would work out. Can anyone help me with it?

Answer 1

To use your function you have to convert tensors and numpy arrays and the other way around. To convert a tensor into a numpy array use tf.eval (see here):

np_array = tensor.eval()

If you want to convert a python object (also numpy array) into tensor you can use tf.convert_to_tensor (see here):

tensor = tf.convert_to_tensor(np.mean(prec),dtype=tf.float32)