TensorFlow Calculate Cosine Distance without NaN Error – TensorFlow Tutorial

Cosine distance is widely used in deep learning,  for example, we can use it to evaluate the similarity of two sentences.

Python Calculate the Similarity of Two Sentences with Gensim

The formula of cosine distance is:

To calculate distance of two vectors, we can use numpy or tensorflow.

Best Practice to Calculate Cosine Distance Between Two Vectors in NumPy

In this tutorial, we will introduce how to calculate it using tensorflow, it avoid tensorflow nan error.

The simplest way to calculate cosine distance is to reference the formula, we can create a function like this:

def cosine(xx, yy):
        x_norm = tf.sqrt(tf.reduce_sum(tf.square(xx), axis=1))
        y_norm = tf.sqrt(tf.reduce_sum(tf.square(yy), axis=1))
        z = tf.reduce_sum(xx*yy,axis=1)
        m = x_norm * y_norm + 1e-9
        cosine = z / m 
        return cosine

Here xx, yy is a 2 dimensions tensors, like [100, 200]. However, this function may occure tensorflow nan error.

The problem is tf.sqrt(x) function will raise a nan value if the value of x is very small.

How to avoid tensorflow nan error?

We shoud do not use tf.sqrt() function.

Here we write a new function to calculate cosine distance in tensorflow.

def cosine(xx, yy):
        x_norm = tf.nn.l2_normalize(xx, axis = 1)
        y_norm = tf.nn.l2_normalize(yy, axis = 1)
        cos = tf.reduce_sum(x_norm * y_norm, axis = 1)
        cosx = tf.clip_by_value(cos, -1.0, 1.0)
        return cosx

Notice: you must use tf.clip_by_value() to limit the value of cos, which may bigger than 1.

How to use this function?

Here is an example, you should create two tensors with 2 dimensions.

import tensorflow as tf
import numpy as np

#create two tensors with 2 dimensions
x = np.array([[2,1,3],[1,1,2]])
y = np.array([[0,1,2],[2,2,2]])
xx = tf.convert_to_tensor(x,dtype=tf.float32)
yy = tf.convert_to_tensor(y,dtype=tf.float32)

Then calculate cosine distance of them by calling cosine(xx, yy) function.

#calculate cosine distance
cos = cosine(xx, yy)

init = tf.global_variables_initializer() 
init_local = tf.local_variables_initializer()

with tf.Session() as sess:
    sess.run([init, init_local])
    print(sess.run([cos]))

Run this code, you will get result like this:

[array([0.83665997, 0.942809  ], dtype=float32)]