Implement Instance Normalization in TensorFlow – TensorFlow Tutorial

Instance Normalization is proposed in paper: Instance Normalization: The Missing Ingredient for Fast Stylization. It also be used in CycleGAN. In this tutorial, we will introduce how to implement it using tensorflow.

Method 1: use tf.contrib.layers.instance_norm()

In tensorflow 1.x, we can use tf.contrib.layers.instance_norm() to implement.

This function is defined as:

tf.contrib.layers.instance_norm(
    inputs, center=True, scale=True, epsilon=1e-06, activation_fn=None,
    param_initializers=None, reuse=None, variables_collections=None,
    outputs_collections=None, trainable=True, data_format=DATA_FORMAT_NHWC,
    scope=None
)

We should notice:

inputs: A tensor with 2 or more dimensions, where the first dimension has batch_size. The normalization is over all but the last dimension if data_format is NHWC and the second dimension if data_format is NCHW

The function is created in normalization.py, we can find:

Method 2: Create a function to implement instance normalization

We also can build a function to implement. Here is an example:

# x is: batch_size * h * w * c
def instance_norm(x):

    with tf.variable_scope("instance_norm"):
        epsilon = 1e-6
        mean, var = tf.nn.moments(x, [1, 2], keep_dims=True)
        scale = tf.get_variable('scale', [x.get_shape()[-1]],
                                initializer=tf.ones_initializer())
        offset = tf.get_variable(
            'offset', [x.get_shape()[-1]],
            initializer=tf.constant_initializer(0.0)
        )
        out = scale * tf.div(x - mean, tf.sqrt(var + epsilon)) + offset

        return out

We can find scale and offset is initialized as tf.contrib.layers.instance_norm(). We will use an example to compare them.

First, we should set a random seed to keep the result stable.

import tensorflow as tf
import random
import os
import numpy as np
import seed_util

seed_util library can be foud in this tutorial:

A Beginner Guide to Get Stable Result in TensorFlow – TensorFlow Tutorial

Then we can use these two methods to compute instance normalization.

inputs = tf.Variable(tf.random_uniform([4, 64, 64, 3], -0.01, 0.01), name = "inputs")

im = tf.contrib.layers.instance_norm(
    inputs
)

im_2 = instance_norm(inputs)
init = tf.initialize_all_variables()
with tf.Session() as sess:
    sess.run(init)
    print(sess.run(im))
    print (sess.run(im_2))

In this code, the inputs is 4*64*64*3. You can regard it as a 4 64*64 images. Run this code, we will get:

tf.contrib.layers.instance_norm
[[[[ 1.2823033   1.1130477   0.7026206 ]
   [-0.68963706  0.85418946 -1.2168176 ]
   [ 0.2991913   1.4322673   0.07365511]
   [ 1.8026736  -1.3883623   0.9475166 ]
   [ 0.8514857   0.25432152 -1.2900805 ]]

  ....

  [[-1.6274862   0.68731815  0.09268501]
   [-0.89866513 -0.84922534 -1.7221147 ]
   [ 1.5203117  -0.0474546  -0.6735875 ]
   [-0.07008501  1.3956294  -0.0356905 ]
   [ 0.09230857 -0.86366445 -0.48223162]]]]
instance_norm
[[[[ 1.2823032   1.1130476   0.7026206 ]
   [-0.689637    0.8541894  -1.2168176 ]
   [ 0.2991913   1.4322673   0.07365511]
   [ 1.8026736  -1.3883623   0.9475166 ]
   [ 0.8514857   0.25432152 -1.2900805 ]]
  ....

  [[-1.6274862   0.68731815  0.09268501]
   [-0.8986652  -0.8492254  -1.7221147 ]
   [ 1.5203117  -0.04745463 -0.67358744]
   [-0.07008501  1.3956294  -0.0356905 ]
   [ 0.09230857 -0.8636645  -0.4822316 ]]]]

From the result, we can find the instance_norm() is same to tf.contrib.layers.instance_norm().