Keras: Method on_batch_end() is slow but I have no callbacks?

Question

Keras just warned me:

UserWarning: Method on_batch_end() is slow compared to the batch update (0.988304). Check your callbacks.

I have no callbacks in my keras script, but I am using DataGenerator (keras.utils.Sequence). Could this be a problem?

For the DataGenerator, I have implemented __init__(), __len__(), __getitem()__, and on_epoch_end methods.

For on_epoch_end, I have:

def on_epoch_end(self):
    """
    This method will be called between every epoch, so we can shuffle the
    indexes here.
    """
    self.indexes = np.arange(len(self.image_names))
    if self.shuffle:
        np.random.shuffle(self.indexes)

The complete callstack is as below:

Using TensorFlow backend. /var/lib/condor/execute/slot1/dir_30551/anaconda/envs/cellimage/lib/python3.6/site-packages/skimage/transform/_warps.py:84: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15.

warn("The default mode, 'constant', will be changed to 'reflect' in " /var/lib/condor/execute/slot1/dir_30551/anaconda/envs/cellimage/lib/python3.6/site-packages/keras/callbacks.py:120: UserWarning: Method on_batch_end() is slow compared to the batch update (0.586719). Check your callbacks. % delta_t_median)

/var/lib/condor/execute/slot1/dir_30551/anaconda/envs/cellimage/lib/python3.6/site-packages/keras/callbacks.py:120: UserWarning: Method on_batch_end() is slow compared to the batch update (0.988304). Check your callbacks. % delta_t_median)

Answer 1

This could be related to your setting for verbose.

I'm seeing the same the same. I believe it's because I have verbose=1 on the call to fit() - i.e. printing the progress bar and current statistics is a batch-end task which is taking a too long. This hypothesis is supported by the evidence that the warnings stop if I set verbose=2 which only prints on epoch end or verbose=0 which doesn't print at all. The warnings also stop if I increase the batch size.