Initialize TensorFlow Weights Using Xavier Initialization : A Beginner Guide – TensorFlow Tutorial

By | July 11, 2020
0 Comment

In order to improve the performance of our mdoel, we can use Xavier method to initialize weights. In this tutorial, we will introduce how to intialize tensorflow weights using Xavier.

What is Xavier?

Xavier is a initialized method, which can keep the scale of the gradients roughly the same in all layers.

Xavier initialization in tensorflow

It contains two forms:

Uniform distribution

Value in weights is in [-x, x]

where x = sqrt(6. / (in + out))

in and out is the shape of weight.

Normal distribution

Value in weights is in [-x, x] and x = sqrt(2. / (in + out))

How to initialize weights using Xavier initialization in tensorflow?

There are two ways: tf.Variable() and tf.get_variable()

We will use an example to show you how to do.

Import library

import tensorflow as tf

Use tf.Variable() to create a weights with Xavier initialization 

w2 = tf.Variable(tf.contrib.layers.xavier_initializer()(([4,4])),name="weights")

Use tf.get_variable() to create a weights with Xavier initialization 

w = tf.get_variable('w', shape=[4, 4])

Output w and w2

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(w))
    print(sess.run(w2))

You will find w and w2 are:

w = 

[[ 0.10678017 -0.54743266 -0.02773023  0.7058236 ]
 [ 0.63566965  0.49337214  0.836279    0.40064913]
 [ 0.6700854   0.24849552 -0.82393456 -0.818665  ]
 [ 0.03686535 -0.68105316  0.73321635  0.15015692]]

w2 = 

[[-0.17646271  0.64760536  0.86489075  0.45141798]
 [ 0.3650096   0.23350495  0.8221863   0.47947305]
 [ 0.07578826  0.6681463  -0.5221854   0.8574583 ]
 [-0.71747375  0.32204002  0.48121613 -0.05668485]]

Leave a Reply