Understand tf.layers.Dense(): How to Use and Regularization – TensorFlow Tutorial

By | March 26, 2021
0 Comment

tf.layers.Dense() is widely used in models built by tensorflow. In this tutorial, we will use some examples to show how to use tf.layers.Dense().

tf.layers.Dense()

tf.layers.Dense() is defined as:

  1. __init__(
  2. units,
  3. activation=None,
  4. use_bias=True,
  5. kernel_initializer=None,
  6. bias_initializer=tf.zeros_initializer(),
  7. kernel_regularizer=None,
  8. bias_regularizer=None,
  9. activity_regularizer=None,
  10. kernel_constraint=None,
  11. bias_constraint=None,
  12. trainable=True,
  13. name=None,
  14. **kwargs
  15. )
__init__(
    units,
    activation=None,
    use_bias=True,
    kernel_initializer=None,
    bias_initializer=tf.zeros_initializer(),
    kernel_regularizer=None,
    bias_regularizer=None,
    activity_regularizer=None,
    kernel_constraint=None,
    bias_constraint=None,
    trainable=True,
    name=None,
    **kwargs
)

It will implement the operation: outputs = activation(inputs * kernel + bias)

Here is an explain:

understand tf.layers.dense() in tensorflow

To understand the structure of tf.layers.Dense(), you can read this tutorial:

Understand Dense Layer (Fully Connected Layer) in Neural Networks – Deep Learning Tutorial

Important Parameters explained

units: dimensionality of the output, for example: 64

activation: can be relu, sigmoid, tanh etc al

use_bias: use bias in activation(inputs * kernel + bias) or not

kernel_initializer: Initializer function for the weight matrix. If None (default), weights are initialized using the default initializer used by tf.get_variable()

Understand How tf.get_variable() Initialize a Tensor When Initializer is None: A Beginner Guide – TensorFlow Tutorial

bias_initializer: Initializer function for the bias.

trainable: Boolean, if True also add variables to the graph collection GraphKeys

name: String, the name of the layer.

reuse: Boolean, whether to reuse the weights of a previous layer by the same name.

How to use tf.layers.Dense()?

We will use an example to show you how to use it.

  1. import tensorflow as tf
  3. x = tf.random_normal([5,3])
  4. y = tf.layers.dense(inputs=x, units=10, activation=tf.nn.relu)
  5. print(y)
  7. with tf.Session() as sess:
  8. sess.run(tf.global_variables_initializer())
  9. print(sess.run(y))
import tensorflow as tf

x = tf.random_normal([5,3])
y = tf.layers.dense(inputs=x, units=10, activation=tf.nn.relu)
print(y)
                    
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(y))

In this example code, the inputs = x, the shape of x is 5*3

units = 10, which means the dimensionality of the output is 5*10.

tf.layers.Dense() will create two tensorflow variables:

w, weight, the shape of it is 3*10

b, bias, the weight of it is 10

Run this code, you will get this result:

y is:

  1. Tensor("dense/Relu:0", shape=(5, 10), dtype=float32)
Tensor("dense/Relu:0", shape=(5, 10), dtype=float32)

The value of y is:

  1. [[0.19549479 0. 0.04906832 0. 0. 0.45005807
  2. 0. 0. 0.10907209 0. ]
  3. [0.16909868 0. 0. 0.5597176 0.06139323 0.8804685
  4. 0.63529086 0. 0.12375151 0. ]
  5. [0.51807237 0. 0.9474018 0.8525198 0. 0.9306468
  6. 1.0625012 0. 0.49360478 1.0925933 ]
  7. [0. 0.68350804 0. 0.9666059 0.8174535 0.
  8. 0.77449316 0.4195258 0. 1.035249 ]
  9. [0. 0.57167256 0. 0. 0.6504928 0.
  10. 0. 0.07965806 0. 0. ]]
[[0.19549479 0.         0.04906832 0.         0.         0.45005807
  0.         0.         0.10907209 0.        ]
 [0.16909868 0.         0.         0.5597176  0.06139323 0.8804685
  0.63529086 0.         0.12375151 0.        ]
 [0.51807237 0.         0.9474018  0.8525198  0.         0.9306468
  1.0625012  0.         0.49360478 1.0925933 ]
 [0.         0.68350804 0.         0.9666059  0.8174535  0.
  0.77449316 0.4195258  0.         1.035249  ]
 [0.         0.57167256 0.         0.         0.6504928  0.
  0.         0.07965806 0.         0.        ]]

How to regularize weights in tf.layers.Dense()?

The simplest way is to get all trainable weights in tf.layers.Dense(). Here is an example:

  1. for n in tf.trainable_variables():
  3. print(n.name)
  4. print(n)
for n in tf.trainable_variables():
    
    print(n.name)
    print(n)

Run this code, you may get this result:

  1. dense/kernel:0
  2. <tf.Variable 'dense/kernel:0' shape=(3, 10) dtype=float32_ref>
  3. dense/bias:0
  4. <tf.Variable 'dense/bias:0' shape=(10,) dtype=float32_ref>
dense/kernel:0
<tf.Variable 'dense/kernel:0' shape=(3, 10) dtype=float32_ref>
dense/bias:0
<tf.Variable 'dense/bias:0' shape=(10,) dtype=float32_ref>

From the result, we can find:

The name of weight is dense/kernel:0 in tf.layers.Dense().

In order to regularize weights in tf.layers.Dense(),  we can read this tutorial:

Multi-layer Neural Network Implements L2 Regularization in TensorFlow – TensorFLow Tutorial

Leave a Reply