Training, Validation, Testing Batch Size Ratio

Question

I'm doing transfer learning using Inception on Tensorflow, this is the training code that I followed : https://raw.githubusercontent.com/tensorflow/hub/master/examples/image_retraining/retrain.py

At the bottom part of the code, we can specify the parameters according to our dataset. (there are training, val, test percentage and training, val, test batch size)
Let's say I have a very large dataset (1 mil) and I already set the training, validation, testing percentage to 75:15:10.

But I have no idea how to set the batch parameters correctly :

• train_batch_size
• validation_batch_size
• test_batch_size

For now, I set the train_batch_size to 64, do I need to set the same value for validation_batch_size? Or should it be bigger or smaller than the train_batch_size?

Answer 1

You can follow the advice from the other answers for the dataset split ratio. However, the batch size has absolutely nothing to do with how you've split your datasets.

The batch size determines how many training examples are processed in parallel for training/inference. The batch size at training time can affect how fast and how well your training converges. You can find a discussion of this effect here. Thus, for train_batch_size, it's worth picking a batch size that is neither too small nor too large (as discussed in the previously linked discussion). For some applications, using the largest possible training batches can actually be desirable, but in general, you select it through experiments and validation.

However, for validation_batch_size and test_batch_size, you should pick the largest batch size that your hardware can handle without running out of memory and crashing. Finding this is usually a simple trial and error process. The larger your batch size at inference time, the faster it will be, since more inputs can be processed in parallel.

EDIT: Here's an additional useful link (Pg. 276) for the training batch size trade-off from Goodfellow et al's deep learning book.