20.5 Introduction to Transformers

The invention of the attention mechanism was a major step forward, but it was still primarily used to augment existing architectures like RNNs. The 2017 paper "Attention Is All You Need" by Vaswani et al. from Google Brain introduced a novel architecture that discarded recurrence and convolution entirely, relying solely on attention. This was the Transformer, and it has become the foundation for nearly all state-of-the-art NLP models, including the Large Language Models (LLMs) we will soon discuss.

The Transformer's key insight is that by using a specific type of attention called self-attention, the model can process all words in a sequence simultaneously, rather than one after another like an RNN. This allows for massive parallelization during training and enables the model to capture complex, long-range dependencies within the text more effectively.

The Transformer Architecture: A High-Level View

Like many sequence-to-sequence models, the original Transformer has an encoder-decoder structure. The encoder's job is to process the input sequence and build a rich representation of it. The decoder's job is to use that representation to generate the output sequence, one element at a time.

Key Components of the Transformer

1. Input Embeddings and Positional Encoding: Like other neural models, the Transformer first converts input tokens into embedding vectors. However, since there is no recurrence, the model has no inherent sense of word order. To solve this, a positional encoding vector is added to each input embedding. This vector provides information about the position of the word in the sequence, allowing the model to understand word order.
2. The Encoder Stack: The encoder is a stack of identical layers (6 in the original paper). Each layer has two main sub-layers:
   • Multi-Head Self-Attention: This is the core of the Transformer. It allows each word in the input sequence to "attend" to all other words in the sequence, calculating a weighted representation of itself based on its relationship with every other word. "Multi-head" means it does this multiple times in parallel with different, learned linear projections, allowing the model to focus on different types of relationships (e.g., one head might focus on syntactic relationships, another on semantic ones).
   • Position-wise Feed-Forward Network: This is a simple, fully connected feed-forward network that is applied to each position's representation independently. It provides additional non-linear processing.
   Each of these sub-layers is followed by a residual connection and layer normalization.
3. The Decoder Stack: The decoder is also a stack of identical layers. It has a similar structure to the encoder but with one crucial addition:
   • Masked Multi-Head Self-Attention: The decoder generates the output sequence one word at a time. During training, we must prevent it from "cheating" by looking at future words in the output sequence it's trying to predict. The self-attention mechanism in the decoder is therefore "masked" to prevent positions from attending to subsequent positions.
   • Encoder-Decoder Attention: This is the second attention layer in the decoder. It allows every position in the decoder to attend to all positions in the encoder's output. This is where the model looks at the input sentence to decide what to output next, similar to the attention mechanism we saw in the previous chapter.
   • Position-wise Feed-Forward Network: Same as in the encoder.
4. Final Linear and Softmax Layer: After the decoder stack, a final linear layer and a softmax function are used to convert the decoder's output into a probability distribution over the vocabulary, from which the next output word is chosen.

Why Was the Transformer So Revolutionary?

• Parallelization: By removing recurrence, the Transformer can process all input tokens in parallel, making training on massive datasets feasible.
• Long-Range Dependencies: Since every word can directly attend to every other word, the path length for information to travel between any two positions is just O(1). In an RNN, this path length is O(N), making it much harder to learn long-range dependencies.
• Model Capacity: The multi-head attention mechanism allows the model to learn a variety of complex relationships between words in a highly flexible way.

The Transformer architecture is the engine that drives modern LLMs like GPT and BERT. In the next chapters, we will explore how these specific models adapt and utilize the Transformer's power for pre-training and fine-tuning on a massive scale.