Understand tf.variable_scope() Reuse Mode for Beginners – TensorFlow Tutorial

By | November 9, 2019
0 Comment

TensorFlow tf.variable_scope() reuse mode is controled by its reuse parameter. In this tutorial, we will discuss how to control reuse model by this reuse parameter for tensorflow beginners.

As to comman way to use tf.variable_scope()

tf.variable_scope(scope_name, reuse)

reuse can assign three values.

None: it means tf.variable_scope() inherit parent variable scope reuse mode, if parent variable scope can reuse variables, it also can.

True: it make variable scope and sub scopes to a reuse mode if sub scopes have set reuse = None.

tf.AUTO_REUSE: it will create variables if they do not exist, and return them otherwise.

When we use tf.variable_scope(), we should use tf.get_variable() function to create or return an existing variable. If you use tf.Variable(), it will create a new variable no matter what value the reuse parameter is.

We will use some examples to illustrate the effect of reuse parameter.

Create two variables in tf.variable_scope()

import tensorflow as tf

with tf.variable_scope('layer1', reuse=tf.AUTO_REUSE):
    w1 = tf.Variable(tf.random_normal(shape=[2,2], mean=0, stddev=1), name='w')
    w2 = tf.get_variable(name = 'w',initializer = tf.random_normal(shape=[3,3], mean=0, stddev=1))

As to this example, reuse=tf.AUTO_REUSE and w1 and w2 are created with same name ‘w‘. w1 = w2?

The answer is w1 ≠ w2.

Output variable name and value

with tf.Session() as sess:  
    sess.run(tf.global_variables_initializer())
    sess.run(tf.local_variables_initializer())
    
    print("w1 = ")
    print(w1.name)
    print(w1.eval())
    print("w2 = ")
    print(w2.name)
    print(w2.eval())

The output is:

w1 =
layer1/w:0
[[-0.03965167 -0.0887058 ]
 [ 1.44239008  0.27801564]]
w2 =
layer1/w_1:0
[[ 0.76585454  0.95292485  0.56786466]
 [-1.31413531 -0.30795729 -0.39146858]
 [-0.17998347 -1.05509949 -0.28664497]]

From the output, we can find, the name of w1 is layer1/w:0, w2 is layer1/w_1:0.

Why w1 ≠ w2, you should know the effect of tf.get_variable().

Understand tf.get_variable(): A Beginner Guide

When reuse = None

As to this example:

import tensorflow as tf

with tf.variable_scope('layer1', reuse= None):
    w1 = tf.get_variable(name = 'w',initializer = tf.random_normal(shape=[3,3], mean=0, stddev=1))
    w2 = tf.get_variable(name = 'w',initializer = tf.random_normal(shape=[3,3], mean=0, stddev=1))

w1 = w2 ?

Run this script, you will get an error:

ValueError: Variable layer1/w already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:

The reason is reuse= None, the parent scope mode is also None, we can not reuse a variable and get an existing variable.

As to reuse = None, it can prohibit tf.get_variable() return an existing variable, it only tf.get_variable() to create a new variable.

Change this example to this.

import tensorflow as tf

with tf.variable_scope('layer1', reuse= tf.AUTO_REUSE):
    with tf.variable_scope('layer_sub_1', reuse = None):
        w1 = tf.get_variable(name = 'w',initializer = tf.random_normal(shape=[3,3], mean=0, stddev=1))
        w2 = tf.get_variable(name = 'w',initializer = tf.random_normal(shape=[3,3], mean=0, stddev=1))

w1 = w2 ?

The answer is yes.

reuse = None in layer_sub_1 scope, however, layer1 scope is tf.AUTO_REUSE. layer_sub_1 scope will inherit the reuse mode of layer1 scope.

tf.variable_scope gets same variables

When reuse = True

Setting resue = True, tf.variable_scope() will get into reuse model.

Look at this example below:

with tf.variable_scope('layer1', reuse= True):
    w1 = tf.get_variable(name = 'w',initializer = tf.random_normal(shape=[3,3], mean=0, stddev=1))
    w2 = tf.get_variable(name = 'w',initializer = tf.random_normal(shape=[3,3], mean=0, stddev=1))

w1 = w2?

Run this code, you will get:

ValueError: Variable layer1/w does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=tf.AUTO_REUSE in VarScope?

Why Variable layer1/w does not exist?

tf.get_variable() can create a new tensorflow only reuse = None or tf.AUTO_REUSE. if reuse = True, tf.get_variable() only can return an existing variable created by tf.get_variable(), can not create a new one.

As to example:

with tf.variable_scope('layer1'):
    w1 = tf.get_variable(name = 'w',initializer = tf.random_normal(shape=[2,2], mean=0, stddev=1))
with tf.variable_scope('layer1', reuse= True):
    w2 = tf.get_variable(name = 'w',initializer = tf.random_normal(shape=[2,2], mean=0, stddev=1))

w1 = w2, because we create a variable with name ‘w‘, then we get this variable with reuse = True.

When reuse = tf.AUTO_REUSE

reuse = tf.AUTO_REUSE is often used in tensorflow application, it create a new variable or return an existing one.

Here is an example:

def layer_weight():
    with tf.variable_scope("weight", reuse = tf.AUTO_REUSE):
        w = tf.get_variable(name = 'w',initializer = tf.random_normal(shape=[2,2], mean=0, stddev=1))
    return w

w1 = layer_weight()
w2 = layer_weight()

In this example, w1 = w2.

Because we create a variable with name ‘w‘, then we get this existing variable.

In general, you should use tf.AUTO_REUSE, it can make you avoid many errors.

Leave a Reply