Initialize torch.nn.Parameter Variable in PyTorch – PyTorch Tutorial

By | May 16, 2022
0 Comment

We may create some parameters in pytorch torch.nn.Module. However, do you know to to initialize it? In this tutorial, we will discuss this topic.

Create a torch.nn.Parameter variable

It is easy to create a torch.nn.Parameter variable in pytorch. For example:

  1. weight = torch.nn.Parameter(torch.randn(5,5))
  2. print(weight)
weight = torch.nn.Parameter(torch.randn(5,5))
print(weight)

Run this code, we will see:

  1. Parameter containing:
  2. tensor([[ 0.0265, -0.0283, 1.4823, -0.2538, 0.4521],
  3. [ 1.0145, 0.3092, 0.9226, -0.3336, 1.0320],
  4. [ 0.3867, -0.2104, -0.1408, -0.6255, -0.3301],
  5. [ 0.9972, -0.9325, -0.3661, -1.9633, -0.3348],
  6. [ 0.6539, -0.3673, -0.3820, -0.5793, 0.6751]], requires_grad=True)
Parameter containing:
tensor([[ 0.0265, -0.0283,  1.4823, -0.2538,  0.4521],
        [ 1.0145,  0.3092,  0.9226, -0.3336,  1.0320],
        [ 0.3867, -0.2104, -0.1408, -0.6255, -0.3301],
        [ 0.9972, -0.9325, -0.3661, -1.9633, -0.3348],
        [ 0.6539, -0.3673, -0.3820, -0.5793,  0.6751]], requires_grad=True)

Initialize torch.nn.Parameter variable with different methods

There are some methods that can initialize torch.nn.Parameter variable.

For example:

  1. import torch
  3. weight = torch.nn.Parameter(torch.Tensor(5, 5))
  4. print(weight)
import torch

weight = torch.nn.Parameter(torch.Tensor(5, 5))
print(weight)

Here we have created a 5*5 empty tensor. It is:

  1. Parameter containing:
  2. tensor([[8.4490e-39, 1.1112e-38, 1.0194e-38, 9.0919e-39, 8.7245e-39],
  3. [1.1112e-38, 4.2245e-39, 8.4489e-39, 9.6429e-39, 8.4490e-39],
  4. [9.6429e-39, 9.2755e-39, 1.0286e-38, 9.0919e-39, 8.9082e-39],
  5. [9.2755e-39, 8.4490e-39, 1.0194e-38, 9.0919e-39, 8.4490e-39],
  6. [1.0745e-38, 1.0653e-38, 1.0286e-38, 1.0194e-38, 9.2755e-39]],
  7. requires_grad=True)
Parameter containing:
tensor([[8.4490e-39, 1.1112e-38, 1.0194e-38, 9.0919e-39, 8.7245e-39],
        [1.1112e-38, 4.2245e-39, 8.4489e-39, 9.6429e-39, 8.4490e-39],
        [9.6429e-39, 9.2755e-39, 1.0286e-38, 9.0919e-39, 8.9082e-39],
        [9.2755e-39, 8.4490e-39, 1.0194e-38, 9.0919e-39, 8.4490e-39],
        [1.0745e-38, 1.0653e-38, 1.0286e-38, 1.0194e-38, 9.2755e-39]],
       requires_grad=True)

In order to initialize it, we can:

Use tensor built-in function

  1. weight.data.uniform_(-1, 1)
  2. print(weight)
weight.data.uniform_(-1, 1)
print(weight)

In this code, we use tensor.uniform_() to initialize the value of weight.

Then we will get:

  1. Parameter containing:
  2. tensor([[-0.1689, 0.4635, -0.6464, -0.9490, -0.6724],
  3. [-0.6435, 0.9929, 0.4329, 0.8690, 0.9690],
  4. [ 0.4082, -0.2569, -0.2214, -0.9469, -0.5764],
  5. [ 0.3985, 0.3738, 0.9745, -0.2259, 0.1704],
  6. [-0.4165, 0.7454, -0.6311, -0.5067, 0.8072]], requires_grad=True)
Parameter containing:
tensor([[-0.1689,  0.4635, -0.6464, -0.9490, -0.6724],
        [-0.6435,  0.9929,  0.4329,  0.8690,  0.9690],
        [ 0.4082, -0.2569, -0.2214, -0.9469, -0.5764],
        [ 0.3985,  0.3738,  0.9745, -0.2259,  0.1704],
        [-0.4165,  0.7454, -0.6311, -0.5067,  0.8072]], requires_grad=True)

Use torch.nn.init function

For example:

  1. torch.nn.init.xavier_uniform_(
  2. weight,
  3. gain=torch.nn.init.calculate_gain("linear"))
  4. print(weight)
torch.nn.init.xavier_uniform_(
           weight,
           gain=torch.nn.init.calculate_gain("linear"))
print(weight)

Here we use torch.nn.init.xavier_uniform_() to initialize the weight.

Understand torch.nn.init.xavier_uniform_() and torch.nn.init.xavier_normal_() with Examples – PyTorch Tutorial

Then we will get:

  1. Parameter containing:
  2. tensor([[ 0.2747, -0.5777, 0.5408, -0.4074, 0.5775],
  3. [ 0.1661, 0.7140, 0.0584, 0.4028, -0.4579],
  4. [-0.5976, -0.3443, 0.5204, -0.2844, -0.5127],
  5. [-0.6429, 0.2265, -0.4873, -0.7025, -0.4240],
  6. [-0.0778, 0.4199, -0.1536, -0.2102, -0.4628]], requires_grad=True)
Parameter containing:
tensor([[ 0.2747, -0.5777,  0.5408, -0.4074,  0.5775],
        [ 0.1661,  0.7140,  0.0584,  0.4028, -0.4579],
        [-0.5976, -0.3443,  0.5204, -0.2844, -0.5127],
        [-0.6429,  0.2265, -0.4873, -0.7025, -0.4240],
        [-0.0778,  0.4199, -0.1536, -0.2102, -0.4628]], requires_grad=True)

Leave a Reply