Orthogonal Regularization is a regularization technique which is often used in convolutional neural networks. In this tutorial, we will introduce it for deep learning beginners.
What is Orthogonal Regularization
There are two types of Orthogonal Regularization, they are:
L1 Norm Orthogonal Regularization
It is defined as:
L2 Norm Orthogonal Regularization
where \(I\) is an identity matrix, \(W\) should be initialized as an orthogonal matrix.
In tensorflow, in order to create a random orthogonal matrix, you can read:
TensorFlow Create a Random Orthogonal Matrix: A Beginner Guide
If you do not use tensorflow, you can read:
Python Create a Random Orthogonal Matrix: A Beginner Guide
In order to compute matrix L1,L2 norm in tensorflow, you can refer:
TensorFlow Calculate Matrix L1, L2 and L Infinity Norm: A Beginner Guide – TensorFlow Tutorial