1.2 Discrete Variables and Distribution

In the previous section, we introduced the concept of a random variable. Now, we will delve deeper into discrete random variables and how their probabilities are described. This is fundamental to LLMs, as the choice of the next word is a discrete random variable selected from the model's vocabulary.

Discrete Random Variables

A discrete random variable is one that can only take on a finite or countably infinite number of distinct values. For example, the number of heads in three coin flips can be 0, 1, 2, or 3. It cannot be 1.5.

In the context of LLMs, the vocabulary is a set of, say, 50,000 words (or tokens). When the model predicts the next word, the random variable representing that word can take on one of these 50,000 discrete values.

Probability Mass Function (PMF)

The Probability Mass Function (PMF) gives the probability that a discrete random variable is exactly equal to some value. If X is a discrete random variable, the PMF is denoted as P(X = x).

A PMF must satisfy two conditions:

1. For any value x, $0 \le P(X=x) \le 1$.
2. The sum of the probabilities over all possible values of x is 1: $\sum_{x} P(X=x) = 1$.

Example: A Fair Six-Sided Die

Let X be the outcome of a roll of a fair die. The possible values for X are {1, 2, 3, 4, 5, 6}. The PMF is:

$P(X=x) = \frac{1}{6}$ for $x \in \{1, 2, 3, 4, 5, 6\}$

Visualization: PMF of a Loaded Die

Below is a visualization of the PMF for a loaded six-sided die. Notice how the probabilities are not equal, but they still sum to 1.

Parametric Visualization

Create an interactive PMF for a 4-sided die. You can adjust the "weight" for each outcome, and the probabilities will automatically re-normalize to ensure they sum to 1. This is analogous to how an LLM adjusts its internal weights during training to refine its output probability distribution.

Connection to LLMs

The output of an LLM for a next-word prediction is exactly a PMF. For a given context (the words that came before), the model computes a probability for every single word in its vocabulary.

If the vocabulary has 50,000 words, the model produces a list of 50,000 probabilities, one for each word.

$P(\text{next_word} = w_i | \text{context})$

The sum of these 50,000 probabilities will be 1. The model then "samples" from this distribution to pick the next word. Words with higher probabilities are more likely to be chosen. This is why understanding discrete distributions is not just related to LLMs—it's their very essence.