Understand tf.variable_scope(): A Beginner Guide – TensorFlow Tutorial

By | November 9, 2019
0 Comment

TensorFlow tf.variable_scope() can create a context manager to manage tensorflow variables in it. We can use it to share variables or create some same name variables. In this tutorial, we will illustrate you how to use it correctly.

Syntax of tf.variable_scope()

tf.variable_scope() is a tensorflow class, not a function.

__init__(
    name_or_scope,
    default_name=None,
    values=None,
    initializer=None,
    regularizer=None,
    caching_device=None,
    partitioner=None,
    custom_getter=None,
    reuse=None,
    dtype=None,
    use_resource=None,
    constraint=None,
    auxiliary_name_scope=True
)

There are many parameters in its initialization function, however, we only focus two parameters:

name_or_scope: the name of scope, we use this name to manage tensorflow variables.

reuse: reuse can be None, True or tf.AUTO_REUSE. It is the most important paramete in tf.variable_scope(). It can control the behaviour of tf.get_variable(). To konw how to control tf.get_variable(), you can read this tutorial.

Understand tf.variable_scope() Reuse Mode for Beginners

In this tutorial, we will discuss two main feature of tf.variable_scope()

1.Make us can share variables

We will create two variables in a context manager.

def layer_weight():
    with tf.variable_scope("weight", reuse = tf.AUTO_REUSE):
        w = tf.get_variable(name = 'w',initializer = tf.random_normal(shape=[2,2], mean=0, stddev=1))
        return w

w1 = layer_weight()
w2 = layer_weight()

We will find w1 = w2, here is output.

with tf.Session() as sess:  
    sess.run(tf.global_variables_initializer())
    sess.run(tf.local_variables_initializer())
    
    print("w1 = ")
    print(w1.name)
    print(w1.eval())
    print("w2 = ")
    print(w2.name)
    print(w2.eval())

The output result is:

tf.variable_scope share variables

So if you want to use this weight in different layers, you can use layer_weight() return a same variable.

2.Create variables with the same name in different scope

Look at this example code:

import tensorflow as tf

with tf.variable_scope('V1'):  
    w1 = tf.get_variable(name='w', shape=[1], initializer=tf.constant_initializer(1))    
with tf.variable_scope('V2'):  
    w2 = tf.get_variable(name='w', shape=[1], initializer=tf.constant_initializer(1))

We create two tensorflow variables in different scope (V1 and V2) with the same name ‘w‘, however, w1 and w2 is not the same, they have different variable name in tensorflow graph.

We output their name and value.

The result is:

w1 =
V1/w:0
[ 1.]
w2 =
V2/w:0
[ 1.]

We find the name of w1 is:V1/w:0, w2 is: V2/w:0. They are two different tensorflow variables.

Leave a Reply