Menu
Tensorflow seems to have a large collection of optimizers, is there any high level guideline (or review paper) on which one is best adapted to specific classes of loss functions ?
968
The Adam optimizer had the best accuracy of 99.2% in enhancing the CNN ability in classification and segmentation.
For most (custom) optimizer implementations, the method apply_gradients() needs to be adapted. This method relies on the (new) Optimizer (class), which we will create, to implement the following methods: _create_slots(), _prepare(), _apply_dense(), and _apply_sparse().
Tensorflow Keras Optimizers Classes:Adagrad: Optimizer that implements the Adagrad algorithm. Adam: Optimizer that implements the Adam algorithm. Adamax: Optimizer that implements the Adamax algorithm. Ftrl: Optimizer that implements the FTRL algorithm.
It depends on your datasets and NN models, but generally, I would start with Adam. Figure 2 in this paper (http://arxiv.org/abs/1412.6980) shows Adam works well.
Also, you can see a very nice animation from http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html.
64
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
