Menu
He / MSRA initialization, from Delving Deep into Rectifiers, seems to be a recommended weight initialization when using ReLUs.
Is there a built-in way to use this in TensorFlow? (similar to: How to do Xavier initialization on TensorFlow)?
450
Xavier initialization is just sampling a (usually Gaussian) distribution where the variance is a function of the number of neurons. tf. random_normal can do that for you, you just need to compute the stddev (i.e. the number of neurons being represented by the weight matrix you're trying to initialize).
From the documentation: If initializer is None (the default), the default initializer passed in the variable scope will be used. If that one is None too, a glorot_uniform_initializer will be used. The glorot_uniform_initializer function initializes values from a uniform distribution.
Initializers define the way to set the initial random weights of Keras layers. The keyword arguments used for passing initializers to layers depends on the layer. Usually, it is simply kernel_initializer and bias_initializer : from tensorflow.keras import layers from tensorflow.keras import initializers layer = layers.
tf.keras.initializers.HeUniform()
or
tf.keras.initializers.HeNormal()
See docs for usage. (h/t to @mable)
tf.contrib.layers.variance_scaling_initializer(dtype=tf.float32)
This will give you He / MRSA initialization. The documentation states that the default arguments for tf.contrib.layers.variance_scaling_initializer correspond to He initialization and that changing the arguments can yield Xavier initialization (this is what is done in TF's internal implementation for Xavier initialization).
Example usage:
W1 = tf.get_variable('W1', shape=[784, 256],
initializer=tf.contrib.layers.variance_scaling_initializer())
or
initializer = tf.contrib.layers.variance_scaling_initializer()
W1 = tf.Variable(initializer([784,256]))
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With