Layer Normalization Explained for Beginners – Deep Learning Tutorial

By | May 24, 2021
0 Comment

Layer Normalization is proposed in paper “Layer Normalization” in 2016, which aims to fix the problem of the effect of batch normalization is dependent on the mini-batch size and it is not obvious how to apply it to recurrent neural networks. In this tutorial, we will introduce what is layer normalization and how to use it.

Layer Normalization

Layer Normalization is defined as:

\(y_i=\lambda(\frac{x_i-\mu}{\sqrt{\sigma^2+\epsilon}})+\beta\)

It is similar to batch normalization. However, as to input \(x\), the normalize axis is different.

Here is an example to normalize the output of BiLSTM using layer normalization.

Normalize the Output of BiLSTM Using Layer Normalization

How to implement layer normalization in tensorflow?

There are two ways to implement:

We will use an example to show you how to do.

import tensorflow as tf
 
x1 = tf.convert_to_tensor(
    [[[18.369314, 2.6570225, 20.402943],
      [10.403599, 2.7813416, 20.794857]],
     [[19.0327, 2.6398268, 6.3894367],
      [3.921237, 10.761424, 2.7887821]],
     [[11.466338, 20.210938, 8.242946],
      [22.77081, 11.555874, 11.183836]],
     [[8.976935, 10.204252, 11.20231],
      [-7.356888, 6.2725096, 1.1952505]]])

mean_x, std_x = tf.nn.moments(x1, axes = 2, keep_dims=True)

v1 = tf.nn.batch_normalization(x1, mean_x, std_x, None, None, 1e-12)
 
v2 = tf.contrib.layers.layer_norm(inputs=x1, begin_norm_axis=-1, begin_params_axis=-1)
with tf.Session() as sess1:
    sess1.run(tf.global_variables_initializer())
    print(sess1.run(v1))
    print(sess1.run(v2))

In this code, v1 is computed by tf.nn.batch_normalization(), v2 is computed by tf.contrib.layers.layer_norm(), we can find the results are the same.

[[[ 0.574993   -1.4064413   0.8314482 ]
  [-0.12501884 -1.1574404   1.2824591 ]]

 [[ 1.3801125  -0.95738953 -0.422723  ]
  [-0.5402142   1.4019756  -0.86176133]]

 [[-0.36398554  1.3654773  -1.0014919 ]
  [ 1.4136491  -0.67222667 -0.7414224 ]]

 [[-1.2645674   0.08396816  1.1806011 ]
  [-1.3146634   1.108713    0.20595042]]]
[[[ 0.574993   -1.4064413   0.8314482 ]
  [-0.12501884 -1.1574404   1.2824591 ]]

 [[ 1.3801125  -0.95738953 -0.422723  ]
  [-0.5402142   1.4019756  -0.86176133]]

 [[-0.36398554  1.3654773  -1.0014919 ]
  [ 1.4136491  -0.67222667 -0.7414224 ]]

 [[-1.2645674   0.08396816  1.1806011 ]
  [-1.3146634   1.108713    0.20595042]]]

As to tf.contrib.layers.layer_norm() source code, we can find:

tf.contrib.layers.layer_norm() calls tf.nn.batch_normalization() to normalize a layer.

https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/contrib/layers/python/layers/layers.py

tf.contrib.layers.layer_norm() implementation

Leave a Reply