How to use TensorFlow metrics in Keras

Question

There seem to be several threads/issues on this already but it doesn't appear to me that this has been solved:

How can I use tensorflow metric function within keras models?

https://github.com/fchollet/keras/issues/6050

https://github.com/fchollet/keras/issues/3230

People seem to either run into problems around variable initialization or the metric being 0.

I need to calculate different segmentation metrics and would like to include tf.metric.mean_iou in my Keras model. This is the best I have been able to come up with so far:

def mean_iou(y_true, y_pred):
   score, up_opt = tf.metrics.mean_iou(y_true, y_pred, NUM_CLASSES)
   K.get_session().run(tf.local_variables_initializer())
   return score

model.compile(optimizer=adam, loss='categorical_crossentropy', metrics=[mean_iou])

This code does not throw any errors but mean_iou always returns 0. I believe this is because up_opt is not evaluated. I have seen that prior to TF 1.3 people have suggested to use something along the lines of control_flow_ops.with_dependencies([up_opt], score) to achieve this. This does not seem possible in TF 1.3 anymore.

In summary, how do I evaluate TF 1.3 metrics in Keras 2.0.6? This seems like quite an important feature.

Answer 1

you can still usecontrol_dependencies

def mean_iou(y_true, y_pred):
   score, up_opt = tf.metrics.mean_iou(y_true, y_pred, NUM_CLASSES)
   K.get_session().run(tf.local_variables_initializer())
   with tf.control_dependencies([up_opt]):
       score = tf.identity(score)
   return score

Answer 2

There were 2 keys to getting this working for me. The first was using

sess = tf.Session()
sess.run(tf.local_variables_initializer())

To initialize TF variables after using the TF functions (and compiling), but before doing model.fit(). You've got that in your initial example, but most other examples show tf.global_variables_initializer(), which didn't work for me.

The other thing I discovered is the op_update object, which is returned as the second part of the tuple from many TF metrics, is what we want. The other portion seems to be 0 when TF metrics are used with Keras. So your IOU metric should look like:

def mean_iou(y_true, y_pred):
   return tf.metrics.mean_iou(y_true, y_pred, NUM_CLASSES)[1]

from keras import backend as K

K.get_session().run(tf.local_variables_initializer())

model.fit(...)