Understand and Calculate Cosine Distance Loss in Deep Learning

Cosine distance loss is often used to object function to evaluate the similarity of vectors in deep learning model. In this tutorial, we will discuss what is cosine distance loss and how to calculate it in deep learning.

What is cosine distance loss

Cosine distance loss is different from cosine distance.

As to cosine distance (cosine) , we can calculate it by numpy or tensorflow.

Best Practice to Calculate Cosine Distance Between Two Vectors in NumPy

TensorFlow Calculate Cosine Distance without NaN Error

As to cosine distance, the value of it:

cosine ∈[-1, 1]

However, the cosine distance loss (cosine_loss) is different, it is equivalent to:

cosine_loss = 1 – cosine

which means cosine_loss ∈[0, 2]

How to use cosine distance loss

We often use cosine distance loss to as loss object function, we should minimize it when training.

How to calculate cosine distance loss

In this tutorial, we will introduce how to calculate it using tensorflow.

In tensorflow, we can use tf.losses.cosine_distance() function to compute it.

tf.losses.cosine_distance(
    labels,
    predictions,
    axis=None,
    weights=1.0,
    scope=None,
    loss_collection=tf.GraphKeys.LOSSES,
    reduction=Reduction.SUM_BY_NONZERO_WEIGHTS,
    dim=None
)

Important parameters

labels, predictions: two tensors we will calculate the cosine distance loss value between them.

axis: The dimension along which the cosine distance is computed

Note:

1.the return value is a 1-D tensor, it is 1- cosine.

2.We should normalize labels and predcitions before using tf.losses.cosine_distance(). To kown how to normalize tensor, you can read this tutorial.

Unit-normalize a TensorFlow Tensor: A Practice Guide

Here we will calculate the cosine distance loss value of two 2-D tensors.

Create two 2-D tensors

These tensors often [batch_zie, length]

import tensorflow as tf
import numpy as np

t1 = tf.Variable(np.array([[1, 4, 5], [5, 5, 7]]), dtype = tf.float32, name = 'lables')
t2 = tf.Variable(np.array([[3, 2, 5], [3, 2, 7]]), dtype = tf.float32, name = 'predictions')

We will calculate cosine distance loss value on axis = 1.

Normalize tensors

t1_norm = tf.nn.l2_normalize(t1, axis = 1)
t2_norm = tf.nn.l2_normalize(t2, axis = 1)

axis = 1 is very important.

Compute cosine distance loss

cosine = tf.losses.cosine_distance(t1, t2, axis = 1)

Then output value

init = tf.global_variables_initializer() 
init_local = tf.local_variables_initializer()

with tf.Session() as sess:
    sess.run([init, init_local])
    print(sess.run([cosine]))

The loss value is: [0.077168673]

Understand and Calculate Cosine Distance Loss in Deep Learning – TensorFlow Tutorial