19.1 Recurrent Neural Networks (RNNs) I: The Core Concept

While CNNs are specialized for spatial data like images, Recurrent Neural Networks (RNNs) are designed to handle sequential data. This includes time series, sentences, audio, and more. Unlike standard feedforward networks, RNNs have loops in them, allowing information to persist.

The Idea of Memory

Imagine reading a sentence. You understand each word based on your understanding of the previous words. A standard neural network processes inputs independently and doesn't have this sense of context. An RNN, on the other hand, maintains a hidden state, which acts as a form of memory.

An RNN cell takes two inputs at each time step \(t\): the input at that step, \(x_t\), and the hidden state from the previous step, \(h_{t-1}\). It then produces two outputs: the output for the current step, \(y_t\), and the new hidden state, \(h_t\), which is passed on to the next time step.

The core recurrence formula is: $$ h_t = f(W_{hh} h_{t-1} + W_{xh} x_t + b_h) $$ $$ y_t = W_{hy} h_t + b_y $$ Where \(f\) is a non-linear activation function (like tanh), and the weight matrices (\(W_{hh}, W_{xh}, W_{hy}\)) and biases (\(b_h, b_y\)) are shared across all time steps. This parameter sharing is what makes RNNs so efficient for sequences of varying lengths.

Unfolding the Recurrence

The loop in an RNN can make it seem complicated. However, we can visualize it by "unfolding" the network through time. This reveals a deep feedforward network where each layer corresponds to a time step, and the weights are shared between layers. This unfolded view is how RNNs are typically implemented and trained.

Interactive RNN Unfolding

This visualization shows a single RNN cell being unfolded over a sequence of inputs. Press the "Next Step" button to feed the next element of the input sequence into the network. Observe how the hidden state from one step is passed to the next, influencing its computation.

Applications

The ability to process sequences makes RNNs suitable for a wide range of tasks:

  • Natural Language Processing (NLP): Language modeling, machine translation, sentiment analysis.
  • Speech Recognition: Converting audio signals into text.
  • Time Series Prediction: Forecasting stock prices, weather, etc.
  • Music Generation: Composing sequences of musical notes.

While simple RNNs are powerful, they struggle with long-term dependencies, a problem we will explore in the next section.