When we are using microsoft neural network intelligence (NNI) for hyper-parameter tuning, we may find the default metric is NaN. In this tutorial, we will introduce a way to fix this error.
What is default metric is NaN?
The NaN error is:
Why does this nan error occur?
There are two main reasons that can cause this error.
1.Do you have used nni.report_intermediate_result() or nni.report_final_result() to save metrics?
If you have used. You can see reason 2. If not, you should use these two functions to save metrics.
2.The data type of metric is right?
For example, as to function:
nni.report_intermediate_result(metrics)
metrics can be any python object. If users use the NNI built-in tuner/assessor, metrics can only have two formats: 1) a number e.g., float, int, or 2) a dict object that has a key named default whose value is a number.
As to our example:
We have used built-in tuner (TPE) and assessor, which means the metrics should be a number or a python dict object that has a key named default.
tuner: builtinTunerName: TPE
Check codes
We have checked our codes and found the reason which make default metric NaN.
Look at code below:
dev_acc = sess.run([accuracy],feed_dict = {x: dev_x, y_label: dev_y, keep_prob_train: 1.0, batch_size_train: dev_x.shape[0]})
nni.report_intermediate_result(dev_acc)
where dev_acc = sess.run([accuracy]) will make dev_acc is a python list, which is not a number or a dict. It can be not processed by python nni.
How to fix this error?
It is very easy to fix this error, we should make dev_acc variable be a number or a dict with a key named ‘default‘.
Here is the solution code.
dev_acc = sess.run(accuracy,feed_dict = {x: dev_x, y_label: dev_y, keep_prob_train: 1.0, batch_size_train: dev_x.shape[0]})
nni.report_intermediate_result(dev_acc)
This code will make dev_acc be a float number.
To know more sess.run() in tensorflow, you can read:
Understand TensorFlow sess.run(): A Beginner Introduction – TensorFlow Tutorial