Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to do weight initialization by Xavier rule in Tensorflow 2.0?

TF 2.0 happened to get rid of contrib library. Therefore, all the goodies like tf.contrib.conv2d or tf.contrib.layers.variance_scaling_initializer are gone. That said, what do you think would be the best way to do Xavier initialization in TF2.0 without using Keras (or initializing with some numpy hack)?

Namely, I am sticking to tf.nn.conv2d and for that function I am the one providing the weights:

filters = tf.Variable(tf.zeros([3, 3, 32, 64]))
??? # something should happen hear, i guess
net = tf.nn.conv2d(input, filters)

Note: Just in case you are using the first version of TF you can just go with:

filters = tf.get_variable("w", shape=[3,3, 32, 64],
           initializer=tf.contrib.layers.xavier_initializer()) 
# no tf.contrib in 2.0, no tf.get_variable in 2.0
like image 507
y.selivonchyk Avatar asked Mar 24 '19 10:03

y.selivonchyk


People also ask

How do I use Xavier initializer in Tensorflow?

Xavier initialization is just sampling a (usually Gaussian) distribution where the variance is a function of the number of neurons. tf. random_normal can do that for you, you just need to compute the stddev (i.e. the number of neurons being represented by the weight matrix you're trying to initialize).

What is the default weight initialization in Tensorflow?

From the documentation: If initializer is None (the default), the default initializer passed in the variable scope will be used. If that one is None too, a glorot_uniform_initializer will be used.

What is kernel initializer in Tensorflow?

Initializers define the way to set the initial random weights of Keras layers. The keyword arguments used for passing initializers to layers depends on the layer. Usually, it is simply kernel_initializer and bias_initializer : from tensorflow.keras import layers from tensorflow.keras import initializers layer = layers.


2 Answers

In tensorflow 2.0 you have a package tf.initializer with all the Keras-like initializers you need.

The Xavier initializer is the same as the Glorot Uniform initializer. Thus, to create a (3,3) variable with values sampled from that initializer you can just:

shape = (3,3)
initializer = tf.initializers.GlorotUniform()
var = tf.Variable(initializer(shape=shape))
like image 188
nessuno Avatar answered Oct 04 '22 13:10

nessuno


Just use glorot uniform initializer which is the same as xavier initializer.

Source: https://www.tensorflow.org/api_docs/python/tf/glorot_uniform_initializer

Also here is an example to prove that they are the same:

tf.reset_default_graph()
tf.set_random_seed(42)
xavier_var = tf.get_variable("w_xavier", shape=[3, 3], initializer=tf.contrib.layers.xavier_initializer())
sess = tf.Session()
sess.run(tf.global_variables_initializer())
print(sess.run(xavier_var))
# [[ 0.27579927 -0.6790426  -0.6128938 ]
#  [-0.49439836 -0.36137486 -0.7235348 ]
#  [-0.23143482 -0.3394227  -0.34756017]]
tf.reset_default_graph()
tf.set_random_seed(42)
glorot_var = tf.get_variable("w_glorot", shape=[3, 3], initializer=tf.glorot_uniform_initializer())
sess = tf.Session()
sess.run(tf.global_variables_initializer())
print(sess.run(glorot_var))
# [[ 0.27579927 -0.6790426  -0.6128938 ]
#  [-0.49439836 -0.36137486 -0.7235348 ]
#  [-0.23143482 -0.3394227  -0.34756017]]

In addition, if you want to the glorot uniform initializer with tf.Variable you can do:

tf.reset_default_graph()
tf.set_random_seed(42)
normal_var = tf.Variable(tf.glorot_uniform_initializer()((3, 3)))
sess = tf.Session()
sess.run(tf.global_variables_initializer())
print(sess.run(normal_var))
# [[ 0.27579927 -0.6790426  -0.6128938 ]
#  [-0.49439836 -0.36137486 -0.7235348 ]
#  [-0.23143482 -0.3394227  -0.34756017]]
like image 31
gorjan Avatar answered Oct 04 '22 13:10

gorjan