Orthogonal Regularization is a regularization technique used in deep learning model. In this tutorial, we will implement it using tensorflow.
What is Orthogonal Regularization?
Orthogonal Regularization is introduced in this tutorial:
Understand Orthogonal Regularization in Deep Learning: A Beginner Introduction
How to implement Orthogonal Regularization using tensorflow?
In this tutorial, we will implement L2 Norm Orthogonal Regularization. Here is an example code:
m2 = tf.matmul(w, w, transpose_b = True) ex = tf.eye(self.class_num) lreg = tf.multiply(m2, (1-ex)) beta = 0.001 loss = beta * tf.nn.l2_loss(lreg)
Here self.class_num is the number of class. Matrix w is self.class_num * m.
loss is the loss value which can be used as a loss function.