Problems must Know Before Building Model based on Memory Networks – Memory Networks Tutorial

What is Memory Networks?

Memory Network is an exciting neural network for deep learning, if you have not know what it is, you can read:

Understand End-To-End Memory Networks – Part 1

Understand End-To-End Memory Networks – Part 2

Although Memory Networks has achieved some exciting results in NLP Task, it also has some problems.

Please look at the output of Memory Networks:

It is linear and may encounter some problems.

Here are two problems:

Problem 1: It supposes u^k and o^k are equally important, however, it will not correct at all time.

So Fei Liu (2017) proposed a gated memory networks (https://www.aclweb.org/anthology/E17-1001) and modified the output of memory networks as:

Problem 2: The output of memory networks is linear, it will not realistic at some time.

Shuai Wang(2018) give us an example in paper “Target-Sensitive Memory Networks for Aspect Sentiment Classification”(https://aclweb.org/anthology/P18-1088).

To aspect sentiment classification, the softmax function is defined:

However, the author provided an example to express the linear out is in conflict in aspect sentiment analysis.

To address this issue, the author provides some nonlinear expressions for softmax, the simplest is:

However, this paper only modified the memory output softmax with nonlinear function, it did not modify each hop output wih nonlinear output, you can try it.

Meanwhile, if you want to use nonlinear function, you can use gated end-to-end memory, which can decrease affection of features that are not related to specific aspect.

Problem 3: Momery networks itself  can not handle time series problems

However, if you want to process time series problems with it, you can read:

Can End-To-End Memory Networks Solve Time Series Problems

To summorize, if you plan to build a neural model based on memory networks, you must notice its output.