Unit-normalize a TensorFlow Tensor: A Practice Guide

Unit-normalize a tensor is a very important tensorflow tip, which can help us to avoid many nan errors. In this tutorial, we will discuss how to normalize a tensorflow tensor for beginners.

What is unit-normalize?

As to tensor or vector V, you can normalize it like this:

It means U is the unit-normalized form of V, the lenth of U is 1.

How to normalized a tensor in tensorflow?

We can use tf.nn.l2_normalize() function to normalize a tensor.

Syntax of tf.nn.l2_normalize

tf.nn.l2_normalize(
    x,
    axis=None,
    epsilon=1e-12,
    name=None,
    dim=None
)

Parameter explained

x: a tensorflow tensor

axis: dimension along which to normalize this tensor x

Here we will write some examples to show you how to use this function correctly.

As to 1-D tensor

We will normalize it(t_norm) and calculate the length (t_norm_len) of its normalized form.

import tensorflow as tf
import numpy as np

t = tf.Variable(np.array([3, 4, 5]), dtype = tf.float32, name = '1-D')
t_norm = tf.nn.l2_normalize(t, axis = 0)
t_norm_len = tf.reduce_sum(tf.square(t_norm))

Then output them.

init = tf.global_variables_initializer() 
init_local = tf.local_variables_initializer()

with tf.Session() as sess:
    sess.run([init, init_local])
    print(sess.run([t_norm, t_norm_len]))

From the result we can find:

t_norm = array([ 0.42426407, 0.56568545, 0.70710683], dtype=float32)
t_norm_len = 1.0

As to 2-D tensor

We will change the axis in tf.nn.l2_normalize() function to calculate. Here is an example.

import tensorflow as tf
import numpy as np

t = tf.Variable(np.array([[3, 4, 5], [5, 6, 7]]), dtype = tf.float32, name = '2-D')
t_norm = tf.nn.l2_normalize(t, axis = 1)
t_norm_len = tf.reduce_sum(tf.square(t_norm), axis = 1)

As to 2-D tensor, we should set axis = 1 in tf.nn.l2_normalize() function.

Then we output t_norm and t_norm_len.

init = tf.global_variables_initializer() 
init_local = tf.local_variables_initializer()

with tf.Session() as sess:
    sess.run([init, init_local])
    print(sess.run([t_norm, t_norm_len]))

The result is:

t_norm = array([[ 0.42426407,  0.56568545,  0.70710683],
       [ 0.4767313 ,  0.57207751,  0.66742378]], dtype=float32)
t_norm = array([ 1.        ,  0.99999988], dtype=float32)

Unit-normalize a TensorFlow Tensor: A Practice Guide – TensorFlow Tips