1.1 Basics of Probability & Random Variables

In our journey towards understanding Large Language Models (LLMs), the first and most crucial step is to grasp the fundamentals of probability. At its core, an LLM is a sophisticated machine that predicts the next word in a sequence. This prediction is not deterministic; it's probabilistic. The model assigns a probability to every possible next word and chooses one based on these probabilities.

What is Probability?

Probability is the measure of the likelihood that an event will occur. It is a number between 0 and 1, where 0 indicates impossibility and 1 indicates certainty.

Mathematically, for a finite sample space S of equally likely outcomes, the probability of an event E is given by:

$$ P(E) = \frac{\text{Number of outcomes in E}}{\text{Total number of outcomes in S}} = \frac{|E|}{|S|} $$

For an LLM, the sample space S is the entire vocabulary of the language, and an event E is the occurrence of a specific word.

Random Variables

A random variable is a variable whose value is a numerical outcome of a random phenomenon. We can think of a random variable as a function that maps outcomes of a random process to numbers.

• Discrete Random Variable: A variable that can take on a finite or countably infinite number of distinct values. For example, the outcome of a dice roll can be {1, 2, 3, 4, 5, 6}. In an LLM, the next word is a discrete random variable, as it's chosen from a finite vocabulary.
• Continuous Random Variable: A variable that can take on any value within a given range. For example, the height of a person.

Interactive Visualization: Coin Flip Experiment

Let's visualize a simple probability experiment: flipping a coin. The outcome is a discrete random variable with two possible values: Heads or Tails. The probability of each is 0.5 for a fair coin.

Flip Coin

Number of Flips: 0

Heads: 0 (Probability: 0.00)

Tails: 0 (Probability: 0.00)

Parametric Visualization

Now, explore how changing a parameter (the bias of a coin) affects the probabilities.

Connection to LLMs

Just like we can calculate the probability of a coin landing on heads, an LLM calculates the probability of the next word appearing in a sentence. For example, after the phrase "The cat sat on the", the LLM might calculate:

• P("mat" | "The cat sat on the") = 0.4
• P("chair" | "The cat sat on the") = 0.2
• P("floor" | "The cat sat on the") = 0.15
• ... and so on for all words in its vocabulary.

The model then uses these probabilities to "choose" the next word. The concepts of sample space, events, and random variables are directly applicable here, forming the mathematical bedrock upon which LLMs are built.