How to get loss for each sample within a batch in keras?

Question

How can I get the loss for each sample within a batch? It seems that Keras does not provide any functions meeting the demand.

Answer 1

Keras always computes losses per sample. It has to do so in order to calculate the loss values that will be used as the basis of back propagation. The values are not typically exposed to the user other than as their average per batch but they are calculated by the loss function and then averaged for display purposes.

A very simple example model:

import tensorflow as tf
import tensorflow.keras.backend as K
keras = tf.keras

model = keras.models.Sequential([
    keras.layers.Input(shape=(4,)),
    keras.layers.Dense(1)
])

def examine_loss(y_true, y_pred):
  result = keras.losses.mean_squared_error(y_true, y_pred)
  result = K.print_tensor(result, message='losses')
  return result

model.compile('adam', examine_loss)
model.summary()

If you execute the follow test code:

import numpy as np
X = np.random.rand(100, 4)

def test_fn(x):
  return x[0] * 0.2 + x[1] * 5.0 + x[2] * 0.3 + x[3] + 0.6

y = np.apply_along_axis(test_fn, 1, X)

model.fit(X[0:4], y[0:4])

You should seem something like the following:

losses [23.2873611 26.1659927 34.1300354 6.16115761]

(Numbers will differ since they depend on random initialisation).

This may interest you or not depending on what you want to do with the individual losses. That was not clear at all from the initial question.

Answer 2

I know this thread is a bit old, but I found a neat solution and maybe this helps someone:

import tensorflow as tf

# Suppose you have an already trained model
model = ...

loss = SparseCategoricalCrossentropy(reduction=tf.compat.v1.losses.Reduction.NONE) # The loss function is just an example, the reduction is the important one
model.compile(optimizer=model.optimizer, loss=loss) # keep your original optimizer

# And then you'll get each loss for each instance within a batch
model.evaluate(X,y, batch_size=128)