20.4 The Attention Mechanism

In our study of Recurrent Neural Networks (RNNs), we saw that they process sequences by maintaining a hidden state that is updated at each time step. For simple tasks, this works well. However, a significant bottleneck exists: the model must compress all the information from the input sequence into a single, fixed-size hidden state vector (often called the "context vector"). This is a huge burden, especially for long sequences. Imagine trying to summarize an entire book in a single paragraph—it's inevitable that you'll lose important details.

The Attention Mechanism, first introduced in the context of machine translation by Bahdanau et al. in 2014, provides an elegant solution to this problem. Instead of relying on a single context vector, attention allows the model to "look back" at the entire input sequence at every step of the output generation process. It learns to selectively focus on the most relevant input words when producing an output word.

The Intuition Behind Attention

Think about how a human translates a sentence. You don't read the whole sentence, memorize it, and then start writing the translation. Instead, as you write each word of the translation, you focus on specific words in the original sentence that are most relevant to the word you're about to write. Attention mimics this process.

In an attention-based model (for example, an RNN decoder in a machine translation system), each output word is generated based on:

1. The previous hidden state of the decoder.
2. A weighted combination of all the hidden states from the encoder (the part of the model that processed the input sentence).

The "weights" in this combination are the key. These are the attention weights, and they are calculated dynamically at each decoding step. A high attention weight for a particular input word's hidden state means that the model is "paying more attention" to that word when generating the current output word.

How Attention is Calculated

The calculation of attention weights generally involves three steps:

1. Scoring Function: For each input hidden state \(h_i\) and the current output hidden state \(s_{t-1}\), we compute a score. A common scoring function is a simple dot product, \( \text{score}(s_{t-1}, h_i) = s_{t-1}^T h_i \), but other functions (like a small feed-forward network) are also used. This score measures how well the input at position \(i\) "matches" the output at the current step.
2. Softmax: The scores are then passed through a softmax function to create a probability distribution. This ensures all the weights are between 0 and 1 and sum to 1. $$ \alpha_{ti} = \frac{\exp(\text{score}(s_{t-1}, h_i))}{\sum_{j=1}^{N} \exp(\text{score}(s_{t-1}, h_j))} $$ These \(\alpha_{ti}\) are the attention weights.
3. Context Vector: A single context vector \(c_t\) is created by taking a weighted sum of all the input hidden states, using the attention weights. $$ c_t = \sum_{i=1}^{N} \alpha_{ti} h_i $$

This context vector \(c_t\), which contains focused information from the input sequence, is then combined with the decoder's hidden state \(s_{t-1}\) to produce the final output for the current time step.

The attention mechanism was a revolutionary idea. It dramatically improved the performance of sequence-to-sequence models and, crucially, it made them more interpretable—we could now "see" what the model was focusing on. This concept of dynamically weighting information would become the single most important component of the Transformer architecture, which we will discuss next.