A Beginner Guide to Get Stable Result in TensorFlow – TensorFlow Tutorial

When you are training or running a tensorflow model, you may find a confused phenomena: The model result may have a little difference during multiple executions, which means the tensorflow model result is not stable. You can not get repeated result.

In this tutorial, we will introduce you how to fix this error and make the tensorflow model result is stable.

Why tensorflow model result is not stable?

The main reason is the weight variables are initialized by some random initializers in tensorflow. These random initializers will generate different values for weight variables in tensorflow.

How to make tensorflow model result is stable?

We should make random initializers generate the same values during different executions. We can set a random seed.

Here is an example:

import tensorflow as tf
import random
import os
import numpy as np
SEED = 1
def set_seeds(seed=SEED):
    os.environ['PYTHONHASHSEED'] = str(seed)
    random.seed(seed)
    tf.set_random_seed(seed)
    np.random.seed(seed)

def set_global_determinism(seed=SEED):
    set_seeds(seed=seed)

    os.environ['TF_DETERMINISTIC_OPS'] = '1'
    os.environ['TF_CUDNN_DETERMINISTIC'] = '1'

    tf.config.threading.set_inter_op_parallelism_threads(1)
    tf.config.threading.set_intra_op_parallelism_threads(1)

In this example code, we have set a SEED for python, numpy and tensorflow random initializers.

Then we can run set_global_determinism() function at the beginner of tensorflow script.

For example, if we create a seed_util.py to save code above, we can run it as follows:

import sys
import datetime
import os, time, pickle
import numpy as np
import tensorflow as tf
import random
import seed_util

g = tf.Graph()
g.seed = seed_util.SEED
with g.as_default():
    seed_util.set_global_determinism()
    session_config = tf.ConfigProto(
        allow_soft_placement=FLAGS.allow_soft_placement,
        log_device_placement=FLAGS.log_device_placement
    )
    session_config.gpu_options.allow_growth = True
    sess = tf.Session(config=session_config, graph=g)

    with sess.as_default():
        # your tensorflow code

Then you will find your tensorflow model result is stable.