Menu
I want my model to run on multiple GPU-sharing parameters but with different batches of data.
Can I do something like that with model.fit()? Is there any other alternative?
520
There are two ways to run a single model on multiple GPUs, data parallelism and device parallelism. In most cases, what you need is most likely data parallelism. Data parallelism consists of replicating the target model once on each device and using each replica to process a different fraction of the input data.
The GPUs can all be on the same node or across multiple nodes. Only gradients are passed between the processes/GPUs. During training, each process loads its own mini-batch from disk and passes it to its GPU. Each GPU does its forward pass, then the gradients are all-reduced across the GPUs.
Strategy is a TensorFlow API to distribute training across multiple GPUs, multiple machines, or TPUs. Using this API, you can distribute your existing models and training code with minimal code changes. tf.
keras models will transparently run on a single GPU with no code changes required.
Keras now has (as of v2.0.9) in-built support for device parallelism, across multiple GPUs, using keras.utils.multi_gpu_model.
Currently, only supports the Tensorflow back-end.
Good example here (docs): https://keras.io/getting-started/faq/#how-can-i-run-a-keras-model-on-multiple-gpus Also covered here: https://datascience.stackexchange.com/a/25737
71
try to use make_parallel function in: https://github.com/kuza55/keras-extras/blob/master/utils/multi_gpu.py (it will work only with the tensorflow backend).
25
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
