What does "num_examples: 2000" mean in TensorFlow object detection config file?

Question

In the sample pipeline config file of TensorFlow object detection, there is this snippet:

  eval_config: {
    num_examples: 2000
    # Note: The below line limits the evaluation process to 10 evaluations.
    # Remove the below line to evaluate indefinitely.
    max_evals: 10
  }

Does "num_examples" mean each evaluation run uses the same first 2000 images, or it treats the test set as a circular buffer and uses different 2000 images each time?

Answer 1

Actually this means only the same top num_examples samples in your evaluation dataset will be used in each run of evaluation.

Answer 2

num_example is equal to the number of test images you are feeding into the API

Answer 3

TL;DR Circular buffer if enough num_epochs and no shuffle

I believe it works in "collaboration" with the input reader config. If in the eval_input_reader you set num_epochs to 1, then it will process the first 2000 images from the input queue, provided the shuffle = false, otherwise some random 2000 images. If you don't have 2000 images, it will probably fail, as the queue is emptied.

The relevant code is here and here