Understand F.cross_entropy(): Compute The Cross Entropy Loss – PyTorch Tutorial

By | July 12, 2022
0 Comment

In pytorch, we can use torch.nn.functional.cross_entropy() to compute the cross entropy loss between inputs and targets. In this tutorial, we will introduce how to use it.

Cross Entropy Loss

It is defined as:

Cross entropy equation 2

This loss often be used in classification problem. The gradient of this loss is here:

Understand the Gradient of Cross Entropy Loss Function – Machine Learning Tutorial

F.cross_entropy()

It is defined as:

  1. torch.nn.functional.cross_entropy(input, target, weight=None, size_average=None, ignore_index=- 100, reduce=None, reduction='mean', label_smoothing=0.0)
torch.nn.functional.cross_entropy(input, target, weight=None, size_average=None, ignore_index=- 100, reduce=None, reduction='mean', label_smoothing=0.0)

input: the input tensor, it can be (N,C) , we should notice: this input tensor is not computed by softmax() function.

target: it usually be an one hot embedding. the shape of it is (N,)

In order to create one hot embedding, you can view:

Understand torch.nn.functional.one_hot() with Examples – PyTorch Tutorial

Here N = batch size.

reduction: it can be none, mean and sum. It determines how to return the loss value. mean is default value.

How to use F.cross_entropy()?

First, we should import some libraries.

  1. import torch
  2. import torch.nn.functional as F
  3. import numpy as np
import torch
import torch.nn.functional as F
import numpy as np

Then we should create an input and target.

  1. input = torch.randn(3, 5, requires_grad=True)
  2. print(input)
  3. target = torch.randint(5, (3,), dtype=torch.int64)
  4. print(target)
input = torch.randn(3, 5, requires_grad=True)
print(input)
target = torch.randint(5, (3,), dtype=torch.int64)
print(target)

Here batch size = 3.

Run this code, we will see:

  1. tensor([[ 1.0491, 0.3516, -1.2480, 0.4829, -1.2766],
  2. [ 1.0018, -0.7298, 0.8515, -0.5951, 1.3111],
  3. [ 0.3726, -0.6266, -1.1043, 0.6200, 0.3317]], requires_grad=True)
  4. tensor([0, 4, 3])
tensor([[ 1.0491,  0.3516, -1.2480,  0.4829, -1.2766],
        [ 1.0018, -0.7298,  0.8515, -0.5951,  1.3111],
        [ 0.3726, -0.6266, -1.1043,  0.6200,  0.3317]], requires_grad=True)
tensor([0, 4, 3])

Then we can start to compute cross entropy loss.

  1. loss = F.cross_entropy(input, target)
  2. print(loss)
  3. loss = F.cross_entropy(input, target, reduction='sum')
  4. print(loss)
  5. loss = F.cross_entropy(input, target, reduction='none')
  6. print(loss)
loss = F.cross_entropy(input, target)
print(loss)
loss = F.cross_entropy(input, target, reduction='sum')
print(loss)
loss = F.cross_entropy(input, target, reduction='none')
print(loss)

Run this code, we will see:

  1. tensor(0.9622, grad_fn=<NllLossBackward>)
  2. tensor(2.8866, grad_fn=<NllLossBackward>)
  3. tensor([0.8170, 0.9723, 1.0973], grad_fn=<NllLossBackward>)
tensor(0.9622, grad_fn=<NllLossBackward>)
tensor(2.8866, grad_fn=<NllLossBackward>)
tensor([0.8170, 0.9723, 1.0973], grad_fn=<NllLossBackward>)

We should notice:

When reduction =  none ,the cross entropy loss of each sample will be returned.

Understand F.cross_entropy() - Compute The Cross Entropy Loss - PyTorch Tutorial

Leave a Reply