LSTMP (LSTM with Recurrent Projection Layer) is an improvement of LSTM with peephole conncections. In this tutorial, we will introduce this model for LSTM Beginners.
Compare LSTMP and LSTM with with peephole conncections
LSTMP is proposed in paper LONG SHORT-TERM MEMORY BASED RECURRENT NEURAL NETWORK ARCHITECTURES FOR LARGE VOCABULARY SPEECH RECOGNITION.
| LSTM with peephole conncections | LSTMP |
 |
 |
\(m_t\) is the output of the current lstm cell. As to equations (1)-(5) and (7)-(11), they are the same.
We usually set \(y_t = m_t\) in LSTM. However, we add a recurrent projection layer in LSTMP.
This recurrent projection layer is:
\[r_t = W_{rm}m_t\]
\(r_t\) is the output of the current lstm cell. We use a weight \(W_{rm}\) to compress or enlarge \(m_t\).
For example:
If the shape of \(m_t\) is 1*200, \(W_{rm}\) is 200* 100. The shape of \(r_t\) is 1* 100. However, if the shape of \(W_{rm}\) is 200* 300. The output of \(r_t\) will be 1 * 300.