Understand How tf.get_variable() Initialize a Tensor When Initializer is None: A Beginner Guide – TensorFlow Tutorial

By | July 12, 2020
0 Comment

Tensorflow tf.get_variable() can create or return an existing tensor, here is the tutorial.

Understand tf.get_variable(): A Beginner Guide – TensorFlow Tutorial

As to tf.get_variable(), it allows initializer = None. Look at the initialized method:

tf.get_variable(
    name,
    shape=None,
    dtype=None,
    initializer=None,
    regularizer=None,
    trainable=True,
    collections=None,
    caching_device=None,
    partitioner=None,
    validate_shape=True,
    use_resource=None,
    custom_getter=None,
    constraint=None
)

Here is a problem: if initializer=None, how tf.get_variable() initialize a new tensor?

Look at the example below:

import tensorflow as tf
w = tf.get_variable('w', shape=[4, 4])
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(w))

Run this code, you will find w is:

[[-0.72751594 -0.7555272  -0.16052002 -0.5544131 ]
 [-0.7395773  -0.23490256  0.11359906 -0.48188818]
 [ 0.19120163 -0.5945381  -0.69641995 -0.6460354 ]
 [-0.33055097 -0.34083188 -0.7476623   0.3036279 ]]

How does tf.get_variable() initialize the value in w?

Look at the source code of tf.get_variable(), we will find the answer.

Source code is here: https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/python/ops/variable_scope.py

If initializer is `None` (the default), the default initializer passed in
the constructor is used. If that one is `None` too, we use a new
`glorot_uniform_initializer`. If initializer is a Tensor, we use
it as a value and derive the shape from the initializer.

We can find: tf.get_variable() will use tf.glorot_uniform_initializer() to initialize the value in w. tf.glorot_uniform_initializer() is the unifrom form of Xavier initialization.

To understand Xavier initialization, you can read this tutorial:

Initialize TensorFlow Weights Using Xavier Initialization : A Beginner Guide

Leave a Reply