18.1 Computer Vision Basics

19.2 Recurrent Neural Networks (RNNs) II: Vanishing/Exploding Gradients and LSTMs

In our previous discussion, we introduced the core concept of Recurrent Neural Networks (RNNs). While powerful, simple RNNs suffer from a significant practical problem that limits their ability to learn long-range dependencies: the vanishing and exploding gradient problem. This chapter explores this issue and introduces a popular and effective solution: the Long Short-Term Memory (LSTM) network.

The Challenge of Long-Range Dependencies

The process of training an RNN involves backpropagation through time (BPTT). As we unfold the network, the gradient of the loss function with respect to the weights at an early time step is calculated by multiplying a sequence of Jacobian matrices. If the values in these matrices are consistently small (less than 1), the gradient signal shrinks exponentially as it propagates back through time, eventually becoming so small that it has no effect on the weight updates. This is the vanishing gradient problem. Conversely, if the matrix values are consistently large (greater than 1), the gradient signal grows exponentially, leading to massive, unstable weight updates. This is the exploding gradient problem.

Mathematically, the gradient at time step $t$ with respect to the hidden state at time step $k$ (where $k < t$) involves a product of derivatives:

$$ \frac{\partial h_t}{\partial h_k} = \prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}} = \prod_{i=k+1}^{t} \mathbf{W}_h^T \text{diag}[f'(h_{i-1})] $$

If the leading eigenvalue of $ \mathbf{W}_h^T $ is less than 1, the product goes to zero (vanishes). If it's greater than 1, it goes to infinity (explodes). Exploding gradients can be mitigated by a technique called gradient clipping, but vanishing gradients are more pernicious, preventing the network from learning correlations between temporally distant events.

Long Short-Term Memory (LSTM)

LSTMs were specifically designed to combat the vanishing gradient problem. They introduce a more complex cell structure that includes a dedicated cell state ($C_t$) and a series of gates that regulate the flow of information. This allows the network to selectively remember or forget information over long periods.

The LSTM Cell Architecture

An LSTM cell has three main gates that control the information flow:

Forget Gate ($f_t$): Decides what information to discard from the cell state. It looks at $h_{t-1}$ and $x_t$ and outputs a number between 0 and 1 for each number in the cell state $C_{t-1}$. A 1 represents "completely keep this" while a 0 represents "completely get rid of this."
Input Gate ($i_t$): Decides which new information to store in the cell state. It has two parts: a sigmoid layer that decides which values to update, and a tanh layer that creates a vector of new candidate values, $\tilde{C}_t$.
Output Gate ($o_t$): Decides what to output from the cell state. The output will be a filtered version of the cell state, passed through a tanh function and multiplied by the output of the sigmoid gate.

The LSTM Update Equations

The gates and states are updated as follows:

Forget Gate:

$$ f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f) $$

Input Gate:

$$ i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i) $$ $$ \tilde{C}_t = \tanh(W_C \cdot [h_{t-1}, x_t] + b_C) $$

Cell State Update:

$$ C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t $$

Output Gate:

$$ o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o) $$ $$ h_t = o_t \odot \tanh(C_t) $$

The key innovation is the cell state $C_t$, which acts like a conveyor belt. Information can flow along it with only minor linear interactions. The gates, controlled by the network's learned weights, can add or remove information, but the default behavior is to preserve the state. This structure makes it much easier for gradients to flow unchanged through many time steps, solving the vanishing gradient problem.

In the next chapter, we will explore another popular RNN variant, the Gated Recurrent Unit (GRU), and discuss practical applications of these powerful sequence models.