How are metrics computed in Keras?

Question

I am wondering how does Keras compute a metric (a custom one or not).

For example, suppose I have the following metric which yields the maximal error between the prediction and the ground truth:

def max_error(y_true, y_pred):

    import keras.backend as K

    return K.max(K.abs(y_true-y_pred))

Is the output scalar metric computed on all mini-batches and then averaged or is the metric directly computed on the whole dataset (training or validation)?

Answer 1

Something additional to know with respect to the metric for the VALIDATION set:

Contrary to what is suggested in another answer, I just saw that the metric on the validation set is calculated in batches, and then averaged (of course the trained model at the end of the epoch is used, in contrast to how the metric score is calculated for the training set).

If you want to compute it on the whole validation data at once, you have to use a callback as described in the accepted answer of guangshengzuo (see https://keras.io/guides/writing_your_own_callbacks/ for more details).

Sure, for the usual metrics, there will not be any difference whether you calculate first in batches and average, or do it all in one big batch. BUT for custom metrics, there very well can be: I just had a case where the metric would tune a parameter, based on the data.

Edit: added link on callbacks, in response to comment