Understand torch.nn.parameter.Parameter() with Examples – PyTorch Tutorial

In this tutorial, we will use some examples to help you understand torch.nn.parameter.Parameter() in pytorch. You can learn how to use it correctly by our examples.

torch.nn.parameter.Parameter()

It is defined as:

torch.nn.parameter.Parameter(data=None, requires_grad=True)

Parameter is the subclass of pytorch Tensor. It is usually used to create some tensors in pytorch Model.

Although we also can use torch.tensor() to create tensors. Here is the tutorial:

4 Methods to Create a PyTorch Tensor – PyTorch Tutorial

However, tensors created by torch.tensor() can not be got by pytorch model parameters() function.

For example:

import torch
import numpy as np

class CustomNN(torch.nn.Module):
    def __init__(self):
       
        super().__init__()
        
        self.c = torch.tensor(np.random.normal(size = [3,3]))
        self.d = torch.randn([3,3])

    def forward(self, x):
        pass

cn = CustomNN()

all_params = cn.named_parameters()

for p in all_params:
    print(p)

In this code, tensors c and d is created by torch.tensor() and torch.randn(). However, we can not get them by cn.named_parameters() or cn.parameters().

tensors created by torch.nn.parameter.Parameter() can be automatically added to the list of pytorch model parameters. We can get them by cn.named_parameters().

For example:

import torch
import numpy as np

class CustomNN(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.a = torch.nn.Parameter(torch.randn(()))
        self.b = torch.nn.Parameter(torch.randn(()))
        self.c = torch.tensor(np.random.normal(size = [3,3]))
        self.d = torch.randn([3,3])

    def forward(self, x):
        pass

cn = CustomNN()

all_params = cn.named_parameters()

for p in all_params:
    print(p)

Run this code, we will see:

('a', Parameter containing:
tensor(1.5615, requires_grad=True))
('b', Parameter containing:
tensor(0.5716, requires_grad=True))

Tensors a and b are got, which is very useful if we plan to implement regularization based on tensor name.