Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

what is the default kernel_initializer in keras

In the user manual, it shows the different kernel_initializer below

https://keras.io/initializers/

the main purpose is to initialize the weight matrix in the neural network.

Anyone knows what the default initializer is? the document didn't show the default.

like image 516
Chris Chou Avatar asked Oct 23 '17 07:10

Chris Chou


People also ask

What is Kernel_initializer keras?

Initializers define the way to set the initial random weights of Keras layers. The keyword arguments used for passing initializers to layers depends on the layer. Usually, it is simply kernel_initializer and bias_initializer : from tensorflow.keras import layers from tensorflow.keras import initializers layer = layers.

What is the default initialization in keras?

The default is glorot initializer. It draws samples from a uniform distribution within [-limit, limit] where limit is sqrt(6 / (fan_in + fan_out)) where fan_in is the number of input units in the weight tensor and fan_out is the number of output units in the weight tensor.

What is the default weight initialization in Tensorflow?

From the documentation: If initializer is None (the default), the default initializer passed in the variable scope will be used. If that one is None too, a glorot_uniform_initializer will be used.


Video Answer


2 Answers

Usually, it's glorot_uniform by default. Different layer types might have different default kernel_initializer. When in doubt, just look in the source code. For example, for Dense layer:

class Dense(Layer): ...     def __init__(self, units,                  activation=None,                  use_bias=True,                  kernel_initializer='glorot_uniform',                  bias_initializer='zeros',                  kernel_regularizer=None,                  bias_regularizer=None,                  activity_regularizer=None,                  kernel_constraint=None,                  bias_constraint=None,                  **kwargs): 
like image 117
Sergey Kovalev Avatar answered Oct 07 '22 22:10

Sergey Kovalev


GlorotUniform, keras uses Glorot initialization with a uniform distribution.r = √(3/fan_avg)

fan_avg = (fan_in + fan_out) /2

number of inputs = fan_in

number of nurons in a layer = fan_out

like image 22
RaaHul Dutta Avatar answered Oct 07 '22 20:10

RaaHul Dutta