batch size on multiple GPUs tensorflow

Question

With tensorflow keras, if I use batch_size=8, and has tf.distribute.MirroredStrategy() with 4 GPUs, for each step in the training, does it has (1) 8 batch, in which each GPU has 2 batch or (2) 4x8 batch, in which each GPU has 8 batch

I thought it is the second case. Please correct me if I am wrong. If it is the second case, how the gradient is calculated across GPUs? By simply average? Does that mean the algorithm and accuracy of 4 GPUs with batch_size=8, equals to a single GPU with batch_size=32? Of course, the training time will be different.

Thank you for your help.

Answer 1

The first interpretation is correct. So if you set batch_size=8. Each GPU has batch size 2 and performs forward and backward passes based on batch size 2. Finally, gradients from 4 GPUs are merged by averaging which is equivalent that batch_size=8 would be processed on single GPU.

Here one can find good exaplanation. https://keras.io/guides/distributed_training/