18.2 Convolutional Neural Networks (CNNs) I

Interactive Visualizations

Explore the fundamental building blocks of CNNs below. Each widget is interactive—adjust parameters to see real-time effects.

1. Convolution Operation Animation

Shows a kernel sliding across an input image while computing outputs.

Kernel Size Stride Padding

2. Feature Map Progression

Illustrates how early layers capture edges and later layers capture abstract concepts.

Depth 1

3. Pooling Strategy Comparison

Compare max, average, and adaptive pooling on the same activation map.

Window Stride

4. Convolution Mathematics

Step through the discrete convolution computation for a single output value.

5. Filter Visualization (Learned Kernels)

Sample learned kernels (synthetic) grouped by layer depth.

Layer Sort

6. Receptive Field Growth Calculator

Compute effective receptive field size across stacked conv/pool layers.

7. CNN Architecture Comparison

Compare layer depth, parameter counts (approx), and innovations of classic architectures.

8. Transfer Learning Visualization

Shows embeddings from source vs fine-tuned layers (synthetic 2D projection).

Layer Show

19.3 Recurrent Neural Networks (RNNs) III: GRUs and Practical Applications

Following our exploration of Long Short-Term Memory (LSTM) networks, we now turn to another popular and powerful variant of the recurrent neural network: the Gated Recurrent Unit (GRU). Introduced by Cho et al. in 2014, the GRU offers a simpler architecture than the LSTM while providing similar performance on many tasks. In this chapter, we'll dissect the GRU cell and then discuss some of the many practical applications of modern RNNs.

The Gated Recurrent Unit (GRU)

The GRU's main objective is to simplify the LSTM's design by combining the forget and input gates into a single "update gate" and merging the cell state and hidden state. This results in a model with fewer parameters, which can be computationally more efficient and require less data to train effectively.

A GRU has two primary gates:

Update Gate ($z_t$): This gate serves a dual purpose, analogous to the forget and input gates of an LSTM. It determines how much of the past information (from the previous hidden state) to keep and how much new information to add.
Reset Gate ($r_t$): This gate determines how to combine the new input with the previous hidden state. It essentially decides how much of the past information to "forget" when creating the new candidate hidden state.

The GRU Cell Architecture

The diagram below illustrates the flow of information within a single GRU cell. Notice the reduced complexity compared to the LSTM cell.

The GRU Update Equations

The gates and the candidate hidden state are calculated as follows:

Reset Gate:

$$ r_t = \sigma(W_r \cdot [h_{t-1}, x_t] + b_r) $$

Update Gate:

$$ z_t = \sigma(W_z \cdot [h_{t-1}, x_t] + b_z) $$

Candidate Hidden State:

$$ \tilde{h}_t = \tanh(W_h \cdot [r_t \odot h_{t-1}, x_t] + b_h) $$

The reset gate $r_t$ is applied to the previous hidden state $h_{t-1}$ before it's used to compute the candidate state $\tilde{h}_t$. This allows the model to effectively ignore parts of the previous state that are irrelevant to the future.

Final Hidden State:

$$ h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t $$

The final hidden state $h_t$ is a linear interpolation between the previous hidden state $h_{t-1}$ and the candidate hidden state $\tilde{h}_t$. The update gate $z_t$ controls this interpolation: if $z_t$ is close to 1, the new state is mostly the candidate state; if it's close to 0, the new state is mostly the previous state, allowing information to pass through unchanged.

Practical Applications of RNNs (LSTMs & GRUs)

The ability of LSTMs and GRUs to model sequential data has led to breakthroughs in numerous fields:

Natural Language Processing (NLP): This is the most common domain for RNNs. Applications include machine translation, sentiment analysis, text generation, question answering, and speech recognition.
Time Series Analysis: Predicting stock prices, weather forecasting, and analyzing sensor data from IoT devices. The model learns temporal patterns to predict future values.
Music Generation: An RNN can be trained on a corpus of musical scores to learn patterns of melody, harmony, and rhythm, and then generate new, original compositions.
Handwriting Recognition: By treating the strokes of a pen as a sequence, RNNs can recognize written characters and words.
Video Analysis: Analyzing the sequence of frames in a video to perform activity recognition or video captioning.

With our understanding of CNNs for spatial data and RNNs for sequential data, we are now equipped to tackle the next major evolution in deep learning: models that combine these ideas to process complex data like text in revolutionary ways. The next chapters will introduce the concept of attention and the Transformer architecture, which form the bedrock of modern Large Language Models.