Keras - method on_batch_end is slow but only callback I have is checkpoint

Question

I set up a network with keras using TensorFlow backend.

When I train my network I often times keep getting message:

UserWarning: Method on_batch_end() is slow compared to the batch update (0.195523). Check your callbacks.
  % delta_t_median)

The issue is that my network is set up with only checkpoint callback:

checkpoint = ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=True, mode='min')
callbacks_list = [checkpoint]

As far as I see in documentation this method is called only on epoch end, so it can't slow down on_batch_end method. Can anyone provide some information on what is the issue?

Answer 1

This is most probably a Generator (fit_generator()) issue. When using a generator as data source it has to be called at the end of a batch. Consider revisiting your generator code, using multiprocessing (workers > 1) or a higher batchsize (if possible)