TF 2.0 happened to get rid of contrib
library. Therefore, all the goodies like tf.contrib.conv2d
or tf.contrib.layers.variance_scaling_initializer
are gone. That said, what do you think would be the best way to do Xavier initialization in TF2.0 without using Keras (or initializing with some numpy hack)?
Namely, I am sticking to tf.nn.conv2d
and for that function I am the one providing the weights:
filters = tf.Variable(tf.zeros([3, 3, 32, 64]))
??? # something should happen hear, i guess
net = tf.nn.conv2d(input, filters)
Note: Just in case you are using the first version of TF you can just go with:
filters = tf.get_variable("w", shape=[3,3, 32, 64],
initializer=tf.contrib.layers.xavier_initializer())
# no tf.contrib in 2.0, no tf.get_variable in 2.0
Xavier initialization is just sampling a (usually Gaussian) distribution where the variance is a function of the number of neurons. tf. random_normal can do that for you, you just need to compute the stddev (i.e. the number of neurons being represented by the weight matrix you're trying to initialize).
From the documentation: If initializer is None (the default), the default initializer passed in the variable scope will be used. If that one is None too, a glorot_uniform_initializer will be used.
Initializers define the way to set the initial random weights of Keras layers. The keyword arguments used for passing initializers to layers depends on the layer. Usually, it is simply kernel_initializer and bias_initializer : from tensorflow.keras import layers from tensorflow.keras import initializers layer = layers.
In tensorflow 2.0 you have a package tf.initializer
with all the Keras-like initializers you need.
The Xavier initializer is the same as the Glorot Uniform initializer. Thus, to create a (3,3)
variable with values sampled from that initializer you can just:
shape = (3,3)
initializer = tf.initializers.GlorotUniform()
var = tf.Variable(initializer(shape=shape))
Just use glorot uniform initializer
which is the same as xavier initializer
.
Source: https://www.tensorflow.org/api_docs/python/tf/glorot_uniform_initializer
Also here is an example to prove that they are the same:
tf.reset_default_graph()
tf.set_random_seed(42)
xavier_var = tf.get_variable("w_xavier", shape=[3, 3], initializer=tf.contrib.layers.xavier_initializer())
sess = tf.Session()
sess.run(tf.global_variables_initializer())
print(sess.run(xavier_var))
# [[ 0.27579927 -0.6790426 -0.6128938 ]
# [-0.49439836 -0.36137486 -0.7235348 ]
# [-0.23143482 -0.3394227 -0.34756017]]
tf.reset_default_graph()
tf.set_random_seed(42)
glorot_var = tf.get_variable("w_glorot", shape=[3, 3], initializer=tf.glorot_uniform_initializer())
sess = tf.Session()
sess.run(tf.global_variables_initializer())
print(sess.run(glorot_var))
# [[ 0.27579927 -0.6790426 -0.6128938 ]
# [-0.49439836 -0.36137486 -0.7235348 ]
# [-0.23143482 -0.3394227 -0.34756017]]
In addition, if you want to the glorot uniform initializer
with tf.Variable
you can do:
tf.reset_default_graph()
tf.set_random_seed(42)
normal_var = tf.Variable(tf.glorot_uniform_initializer()((3, 3)))
sess = tf.Session()
sess.run(tf.global_variables_initializer())
print(sess.run(normal_var))
# [[ 0.27579927 -0.6790426 -0.6128938 ]
# [-0.49439836 -0.36137486 -0.7235348 ]
# [-0.23143482 -0.3394227 -0.34756017]]
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With