1.3 Common Discrete Distributions

While the Probability Mass Function (PMF) can describe any discrete distribution, some distributions are so common and useful that they have been given special names. Understanding these is crucial as they form the building blocks for more complex models, including those used in LLMs.

1. Bernoulli Distribution

The Bernoulli distribution is the simplest. It represents a single trial with two possible outcomes: success (1) or failure (0). The probability of success is denoted by p.

PMF: $ P(X=k) = p^k (1-p)^{1-k} $ for $k \in \{0, 1\} $

LLM Connection: At a very basic level, you can think of a Bernoulli trial in language as: "Is the next word 'king'?" If the model's vocabulary has 50,000 words, this is one trial out of many possibilities.

2. Binomial Distribution

The Binomial distribution models the number of successes in a fixed number, n, of independent Bernoulli trials, each with the same probability of success, p.

PMF: $ P(X=k) = \binom{n}{k} p^k (1-p)^{n-k} $ where $ \binom{n}{k} = \frac{n!}{k!(n-k)!} $

LLM Connection: Imagine a document with n words. If we assume the probability of any given word being "transformer" is p, the Binomial distribution could model how many times the word "transformer" appears in the document.

3. Poisson Distribution

The Poisson distribution models the number of events occurring in a fixed interval of time or space, given an average rate of occurrence, $ \lambda $ (lambda).

PMF: $ P(X=k) = \frac{\lambda^k e^{-\lambda}}{k!} $

LLM Connection: The frequency of rare words in a large text corpus often follows a Poisson distribution. For example, the number of times the word "serendipity" appears per 10,000 pages of text can be modeled with a Poisson distribution.

---

Interactive Visualization

Use the controls below to explore these distributions. Adjust parameters and observe how the chart responds.