Keras early stopping callback error, val_loss metric not available

Question

I am training a Keras (Tensorflow backend, Python, on MacBook) and am getting an error in the early stopping callback in fit_generator function. The error is as follows:

RuntimeWarning: Early stopping conditioned on metric `val_loss` which is not available. Available metrics are:
  (self.monitor, ','.join(list(logs.keys()))),
RuntimeWarning: Can save best model only with val_acc available, skipping.

'skipping.' % (self.monitor), RuntimeWarning
[local-dir]/lib/python3.6/site-packages/keras/callbacks.py:497: RuntimeWarning: Early stopping conditioned on metric `val_loss` which is not available. Available metrics are:
  (self.monitor, ','.join(list(logs.keys()))), RuntimeWarning
[local-dir]/lib/python3.6/site-packages/keras/callbacks.py:406: RuntimeWarning: Can save best model only with val_acc available, skipping.
  'skipping.' % (self.monitor), RuntimeWarning)
Traceback (most recent call last):
  :
  [my-code]
  :
  File "[local-dir]/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
  File "[local-dir]/lib/python3.6/site-packages/keras/engine/training.py", line 2213, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
  File "[local-dir]/lib/python3.6/site-packages/keras/callbacks.py", line 76, in on_epoch_end
callback.on_epoch_end(epoch, logs)
  File "[local-dir]/lib/python3.6/site-packages/keras/callbacks.py", line 310, in on_epoch_end
self.progbar.update(self.seen, self.log_values, force=True)
AttributeError: 'ProgbarLogger' object has no attribute 'log_values'

My code is as follows (which looks OK):

:
ES = EarlyStopping(monitor="val_loss", min_delta=0.001, patience=3, mode="min", verbose=1)
:
self.model.fit_generator(
        generator        = train_batch,
        validation_data  = valid_batch,
        validation_steps = validation_steps,
        steps_per_epoch  = steps_per_epoch,
        epochs           = epochs,
        callbacks        = [ES],
        verbose          = 1,
        workers          = 3,
        max_queue_size   = 8)

The error message appears to relate to the early stopping callback but the callback looks OK. Also the error states that the val_loss is not appropriate, but I am not sure why... one more unusual thing about this is that the error only occurs when I use smaller data sets.

Any help is appreciated.

Answer 1

If the error only occurs when you use smaller datasets, you're very likely using datasets small enough to not have a single sample in the validation set.

Thus it cannot calculate a validation loss.

Answer 2

The error occurs to us because we forgot to set validation_data in fit() method, while used 'callbacks': [keras.callbacks.EarlyStopping(monitor='val_loss', patience=1)],

Code causing error is:

self.model.fit(
        x=x_train,
        y=y_train,
        callbacks=[keras.callbacks.EarlyStopping(monitor='val_loss', patience=1)],
        verbose=True)

Adding validation_data=(self.x_validate, self.y_validate), in fit() fixed:

self.model.fit(
        x=x_train,
        y=y_train,
        callbacks=[keras.callbacks.EarlyStopping(monitor='val_loss', patience=1)],
        validation_data=(x_validate, y_validate),
        verbose=True)

Answer 3

I up-voted the previous answer as it gave me the insight to verify the data and inputs to the fit_generator function and find out what the root cause of the issue actually was. In summary, in cases where my dataset was small, I calculated validation_steps and steps_per_epoch which turned out to be zero (0) which caused the error.

I suppose the better longer-term answer, perhaps for the Keras team, is to cause an error/exception in fit_generator when these values are zero, which would probably lead to a better understanding about how to address this issue.