As to batch normalization in machine learning, there are two important points we must concern:
- How to compute the mean and variance of population sample by train batch sample?
- How to normalize the test batch sample by the mean and variance of population sample?
In this tutorial, we will introduce you how to do.
How to compute the mean and variance of population sample by train batch sample?
In order to compute the mean and variance of population sample, we should compute the mean and variance of train batch sample.
Here is a tutorial to help you understand how to compute.
Understand the Mean and Variance Computed in Batch Normalization – Machine Learning Tutorial
In tensorflow, we can use tf.nn.moments() to compute.
Calculate the Mean and Variance of a Tensor in TensorFlow – TensorFlow Tutorial
Here is an example code:
batch_mean, batch_var = tf.nn.moments(x,[0])
First, we set the population sample mean and variance to 0. It means:
pop_mean_op = 0.0 pop_var_op = 0.0
Then we will define a decay (such as 0.99) to update pop_mean_op and pop_var_op.
decay = 0.99
Then we can use equation below to update pop_mean_op and pop_var_op.
pop_op = decay * pop_op + (1 - decay) * batch_op
Here is an example code:
pop_mean_op = tf.assign(pop_mean, pop_mean * decay + batch_mean * (1 - decay)) pop_var_op = tf.assign(pop_var, pop_var * decay + batch_var * (1 - decay))
Finally, we can get the mean and variance of population sample by train batch sample.
Notice
We only can update pop_mean_op and pop_var_op when training model, if you are using model to compute the accuracy based on test dataset, you can not update pop_mean_op and pop_var_op.
How to normalize the test batch sample by the mean and variance of population sample?
In order to normalize the test batch sample, we only can use the mean and variance of population sample (pop_mean_op, pop_var_op), we can not use the mean and variance of test sample.
In tensorflow, we can use tf.nn.batch_normalization() to normalize the test batch sample.