How to correctly use the Tensorflow MeanIOU metric?

Question

I want to use the MeanIoU metric in keras (doc link). But I don't really understand how it could be integrated with the keras api. In the example, the prediction and the ground truth are given as binary values but with keras we should get probabilities, especially because the loss is mse... We should have something like:

m = tf.keras.metrics.MeanIoU(num_classes=2)
m.update_state([0, 0, 1, 1], [0.3, 0.6, 0.2, 0.9])

But now the result isn't the same, we have:

# <tf.Variable 'UnreadVariable' shape=(2, 2) dtype=float64, numpy=array([[2., 0.],
#                                                                        [2., 0.]])>
m.result().numpy() # 0.25

So my question is how should we use this metric if the output of the model is probabilities? binary or even in a multi-class setting (one hot)?

For the Accuracy there is a distinction between BinaryAccuracy and CategoricalAccuracy and they both take probabilities in y_pred. Shouldn't it be the same for MeanIoU?

Answer 1

I am having similar issues. Despite looking for examples online, all demonstrations happens after applying argmax on the model's output.

The workaround I have for now is to subclass tf.keras.metrics.MeanIoU:

class MyMeanIOU(tf.keras.metrics.MeanIoU):
    def update_state(self, y_true, y_pred, sample_weight=None):
        return super().update_state(tf.argmax(y_true, axis=-1), tf.argmax(y_pred, axis=-1), sample_weight)

It is also possible to create your own function, but it is recommended to subclass tf.keras.metrics.Metric if you wish to benefit from the extra features such as distributed strategies.

I am still looking for cleaner solutions.

Answer 2

i have the same problem, and i look into the source code.

In tf2.0, at the end of the update_state function, there is :

current_cm = confusion_matrix.confusion_matrix(
        y_true,
        y_pred,
        self.num_classes,
        weights=sample_weight,
        dtype=dtypes.float64)

and i look into confusion_matrix function,

with ops.name_scope(name, 'confusion_matrix',
                      (predictions, labels, num_classes, weights)) as name:
    labels, predictions = remove_squeezable_dimensions(
        ops.convert_to_tensor(labels, name='labels'),
        ops.convert_to_tensor(
            predictions, name='predictions'))
    predictions = math_ops.cast(predictions, dtypes.int64)
    labels = math_ops.cast(labels, dtypes.int64)

    # Sanity checks - underflow or overflow can cause memory corruption.
    labels = control_flow_ops.with_dependencies(
        [check_ops.assert_non_negative(
            labels, message='`labels` contains negative values')],
        labels)
    predictions = control_flow_ops.with_dependencies(
        [check_ops.assert_non_negative(
            predictions, message='`predictions` contains negative values')],
        predictions)

    if num_classes is None:
      num_classes = math_ops.maximum(math_ops.reduce_max(predictions),
                                     math_ops.reduce_max(labels)) + 1
    else:
      num_classes_int64 = math_ops.cast(num_classes, dtypes.int64)
      labels = control_flow_ops.with_dependencies(
          [check_ops.assert_less(
              labels, num_classes_int64, message='`labels` out of bound')],
          labels)
      predictions = control_flow_ops.with_dependencies(
          [check_ops.assert_less(
              predictions, num_classes_int64,
              message='`predictions` out of bound')],
          predictions)

    if weights is not None:
      weights = ops.convert_to_tensor(weights, name='weights')
      predictions.get_shape().assert_is_compatible_with(weights.get_shape())
      weights = math_ops.cast(weights, dtype)

    shape = array_ops.stack([num_classes, num_classes])
    indices = array_ops.stack([labels, predictions], axis=1)
    values = (array_ops.ones_like(predictions, dtype)
              if weights is None else weights)
    cm_sparse = sparse_tensor.SparseTensor(
        indices=indices,
        values=values,
        dense_shape=math_ops.cast(shape, dtypes.int64))
    zero_matrix = array_ops.zeros(math_ops.cast(shape, dtypes.int32), dtype)

    return sparse_ops.sparse_add(zero_matrix, cm_sparse)

the trick is at 6th line of the code, tf use math_ops.cast cast the predictions to int64, so when you send [0.3, 0.6, 0.2, 0.9] into cast function, it returns [0, 0, 0, 0].

So, that's why you got a confusion maxtrix

[[2., 0.],
[2., 0.]]