Understand Adaptive Gating Mechanism in Deep Learning – Deep Learning Tutorial

Adaptive Gating Mechanism can dynamically control the information flow based on the current input, which often be a sigmoid function.

In LSTM

In gated end-to-end memory networks

There are some important questions you should know to understand Adaptive Gating Mechanism.

Q l: What is gate?

Gate is often a sigmoid function, value of it in (0,1)

Q 2: How to understand adaptive?

Adaptive means the value of gate is trained by model, the value of each gate (sigmoid) is determined by inputs and trainable variables. So different inputs will generate different value, which is adaptive.

Q 3: How to control information flow by gate?

The value of each gate is in (0,1), if the value is equal to 0, the information will be quite.

Q4: How to use gate mechanism?

Gate mechanism often be used in two inputs, such as x_t and h_t-1 or o^k and u^k, we often use gate to evaluate the different weight of each input.

But if the number of input is better than two, gate mechanism will not wok, we will need use attention mechanim to evaluate the different weight of each input.