Understand torch.nn.Dropout() with Examples – PyTorch Tutorial

By | May 17, 2022
0 Comment

torch.nn.Dropout() allows us to create a dropout layer easily in pytorch. In this tutorial, we will use some examples to show you how to use it.

torch.nn.Dropout()

It is defined as:

torch.nn.Dropout(p=0.5, inplace=False)

It will randomly zero some of the elements of the input tensor with probability p using samples from a Bernoulli distribution during training. However, it will not work during model evaluation mode.

We should notice: the outputs are scaled by a factor of \(\frac{1}{1-p}\)during training, factor will be an identity function during evaluation.

Parameters

p: probability of an element to be zeroed. Default: 0.5

inplace: If set to True, will do this operation in-place. Default: False

We will use some examples to show you how to use this function.

torch.nn.Dropout() examples

Look at the example below:

import torch
import torch.nn as nn

x = torch.randn(5,5)
print(x)
dropout = torch.nn.Dropout(0.4)
y = dropout(x)
print(y)
print(x)

Run this code, we will find:

The original x is:

tensor([[ 1.5510,  0.3748,  0.1169, -0.4646,  0.1686],
        [-0.0432, -1.3270, -0.6264,  0.3724, -0.1386],
        [-0.1403, -0.6082,  0.4403,  0.8334, -0.6324],
        [ 1.7417,  0.0482, -1.3543, -1.5202,  0.1210],
        [-1.2560, -1.0618, -0.1343,  0.3552, -0.6313]])

Passing it to a dropout with p = 0.4, we can find y is:

tensor([[ 2.5850,  0.0000,  0.1948, -0.7744,  0.2810],
        [-0.0720, -2.2116, -1.0439,  0.0000, -0.2310],
        [-0.0000, -1.0137,  0.7338,  1.3890, -1.0540],
        [ 2.9029,  0.0804, -2.2571, -2.5337,  0.0000],
        [-2.0933, -1.7697, -0.0000,  0.0000, -1.0521]])

We can find 40% elements are set to 0 in x.

However, as to 1.5510 in x, it is set to 2.585. Why?

Because p = 0.4, 1.5510 * 1 / (1 – 0.4) = 1.5510 * 5 / 3 = 2.585

In this example, inplace = False, which means x is not changed.

However, if inplace = True. For example:

import torch
import torch.nn as nn

x = torch.randn(5,5)
print(x)
dropout = torch.nn.Dropout(0.4, inplace=True)
y = dropout(x)
print(y)
print(x)

Run this code, we will see:

tensor([[ 0.5089,  0.7897,  1.3041, -1.5822,  0.5035],
        [ 0.2674, -0.3560, -0.7442,  1.9541,  0.1558],
        [ 0.1353,  0.7813,  0.5371, -0.9287,  0.7181],
        [-0.3352, -0.3901, -0.1539, -0.5311,  0.6983],
        [-0.2276, -0.7470,  0.2549,  1.5447,  0.4305]])
tensor([[ 0.8482,  1.3162,  0.0000, -0.0000,  0.0000],
        [ 0.0000, -0.5933, -1.2403,  3.2568,  0.0000],
        [ 0.2255,  1.3022,  0.8952, -0.0000,  1.1968],
        [-0.0000, -0.0000, -0.2565, -0.0000,  1.1638],
        [-0.0000, -1.2450,  0.0000,  0.0000,  0.7175]])
tensor([[ 0.8482,  1.3162,  0.0000, -0.0000,  0.0000],
        [ 0.0000, -0.5933, -1.2403,  3.2568,  0.0000],
        [ 0.2255,  1.3022,  0.8952, -0.0000,  1.1968],
        [-0.0000, -0.0000, -0.2565, -0.0000,  1.1638],
        [-0.0000, -1.2450,  0.0000,  0.0000,  0.7175]])

We can find x is changed, x = y when inplace = True.

torch.nn.Dropout() can work during training. We can control it effect by setting model mode. Here is the tutorial:

An Introduction to PyTorch model.eval() for Beginners – PyTorch Tutorial

Leave a Reply