Keras evaluate_generator with callback

Question

I have a keras model and I try to test it with test data and the evaluate_generator method. I have a use case where a callback in this method would come in handy. In the keras docs: evaluate_generator there is a callback argument. However when I test this with following code I get an error.

model = load_model('/models/model.h5')

img_width = 120
img_height = 120
batch_size = 32

data_generator = ImageDataGenerator(rescale=1. / 255.)

test_generator = data_generator.flow_from_directory(
        '/data/test',
        target_size=(img_width, img_height),
        batch_size=batch_size,
        class_mode='binary')

STEP_SIZE_TEST = test_generator.n // test_generator.batch_size


class TestCallback(Callback):
    def on_test_batch_begin(self, batch, logs=None):
        print('Evaluating batch: ' + str(batch))


test_callback = TestCallback()

model.evaluate_generator(test_generator, STEP_SIZE_TEST, callbacks=[test_callback])

The error:

Traceback (most recent call last):
  File "/test_callback.py", line 34, in <module>
    model.evaluate_generator(generator=test_generator, steps=STEP_SIZE_TEST, callbacks=[test_callback])
  File "/miniconda3/envs/models/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
TypeError: evaluate_generator() got an unexpected keyword argument 'callbacks'

When I edit the code and leave out the keyword like so:

model.evaluate_generator(test_generator, STEP_SIZE_TEST, [test_callback])

I get following error:

Exception in thread Thread-16:
Traceback (most recent call last):
  File "/miniconda3/envs/models/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/miniconda3/envs/models/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/miniconda3/envs/models/lib/python3.6/site-packages/keras/utils/data_utils.py", line 671, in _run
    executor.apply_async(next_sample, (self.uid,)), block=True)
  File "/miniconda3/envs/models/lib/python3.6/queue.py", line 127, in put
    if self.maxsize > 0:
TypeError: '>' not supported between instances of 'list' and 'int'

My keras version is 2.2.4

Answer 1

The documentation you're seeing is for master branch. The 'callbacks' argument is not supported on 2.2.4. In 2.2.4, the third argument is still max_queue_size and hence there is an error when interpreting the [test_callback] list as an int.