Understand tf.nn.softmax_cross_entropy_with_logits_v2(): A Beginner Guide

TensorFlow tf.nn.softmax_cross_entropy_with_logits_v2() is one of functions which tensorflow use to compute cross entropy, which is very similar to tf.nn.softmax_cross_entropy_with_logits(). In this tutorial, we will introduce how to use this function for tensorflow beginners.

Syntax

tf.nn.softmax_cross_entropy_with_logits_v2(
    _sentinel=None,
    labels=None,
    logits=None,
    dim=-1,
    name=None
)

Computes softmax cross entropy between logits and labels

Parameter explained

labels: it should be a valid probability distribution, which means sum(labels) = 1

logits: a probability, it can not be computed by tf.nn.softmax() first, because it will be computed by tf.nn.softmax() in this function.

Notices you must remmeber

There are some notices you should concern when you are using this function.

1.logits can not be computed by softmax operation first

2.the value of labels is also can be update when training, which means you must set tf.stop_gradient(labels) if you use this function in classification problem.

Here we will use an example to explain how to use this function.

Create labels and logits tensor

import tensorflow as tf

logits = tf.Variable(np.array([[1, 2, 3],[4, 5, 6]]), dtype = tf.float32)
labels = tf.Variable(np.array([[1, 0, 0],[, 1, 0]]), dtype = tf.float32)

In this code, we have created labels and logits with 2 * 3 shape.

Calculate cross entropy with labels and logits

cross_entropy_loss=tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits = logits, labels = labels))

Output cross entropy

init = tf.global_variables_initializer() 
init_local = tf.local_variables_initializer()

with tf.Session() as sess:
    sess.run([init, init_local])
    print(sess.run([cross_entropy_loss]))

The cross entropy value is: 1.9076059

We also can use our customized cross entropy function to get this value:

logits_softmax = tf.nn.softmax(logits, axis = 1)
loss= compute_cross_entropy(logits = logits_softmax , labels = labels)

The loss value is also 1.9076059.

Why this function update labels

Labels tensor are usually constant in classification problem, we need not update value when training. However, we also can use some models to create labels, such as p and q, they are two distributions generated by some models, if we want to calculate the cross entropy between them, we can use this function.

cross_entropy_loss=tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits = p, labels = q))

Which means p and q can be updated when training.

Understand tf.nn.softmax_cross_entropy_with_logits_v2(): A Beginner Guide – TensorFlow Tutorial