2.2 Common Continuous Distributions
Many continuous distributions appear repeatedly in machine learning theory and in the internal statistical behavior of Large Language Models (LLMs). Understanding parameter effects sharpens intuition about optimization stability, regularization, and uncertainty.
1. Normal (Gaussian) Distribution
PDF: $ f(x;\mu,\sigma^2) = \dfrac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}} $. Symmetric, fully characterized by mean $\mu$ and variance $\sigma^2$. Appears via Central Limit Theorem and in weight initialization assumptions.
2. Exponential Distribution
PDF: $ f(x;\lambda) = \lambda e^{-\lambda x}$ for $x \ge 0$. Memoryless. Models waiting times; analogous to spacing between rare token categories in corpora.
3. Beta Distribution
Support: $x \in (0,1)$. PDF: $ f(x;\alpha,\beta) = \dfrac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)} x^{\alpha-1} (1-x)^{\beta-1} $. Flexible shapes for probabilities. In LLM pretraining, Beta-like shapes can approximate token proportion uncertainties.
4. Uniform Distribution
PDF: $ f(x;a,b) = \dfrac{1}{b-a}$ for $a \le x \le b$. Maximum entropy given bounded support and fixed interval. Useful as a neutral prior over ranges.
5. Student's t Distribution (Bonus)
PDF (degrees of freedom $\nu$): $ f(x;\nu) = \dfrac{\Gamma((\nu+1)/2)}{\sqrt{\nu\pi}\,\Gamma(\nu/2)} \left(1+\dfrac{x^2}{\nu}\right)^{-(\nu+1)/2} $. Heavier tails than Normal; relevant to robustness (mitigates influence of outliers).
Moments (Selected)
- Normal: Mean $=\mu$, Var $=\sigma^2$.
- Exponential: Mean $=1/\lambda$, Var $=1/\lambda^2$.
- Beta: Mean $=\alpha/(\alpha+\beta)$, Var $= \dfrac{\alpha\beta}{(\alpha+\beta)^2(\alpha+\beta+1)}$.
- Uniform: Mean $(a+b)/2$, Var $(b-a)^2/12$.
- Student t: Mean $=0$ for $\nu>1$; Var $= \nu/(\nu-2)$ for $\nu>2$.
Interactive Multi-Distribution Plot
Toggle distributions and adjust parameters. The plot rescales automatically. Beta supports rich shapes; try extreme $\alpha, \beta$ values. Observe how tail heaviness (Student t with low $\nu$) compares to Normal—a concept linked to gradient noise robustness in training.
Visibility
Normal Parameters
μ: {{params.normal.mu | number:2}}σ: {{params.normal.sigma | number:2}}
Exponential λ
λ: {{params.exponential.lambda | number:2}}Beta Parameters
α: {{params.beta.alpha | number:2}}β: {{params.beta.beta | number:2}}
Uniform Interval
a: {{params.uniform.a | number:2}}b: {{params.uniform.b | number:2}}
Student t ν
ν: {{params.student.nu | number:0}}Point Evaluation
x: {{probe.x | number:2}}| Distribution | f(x) |
|---|---|
| {{row.name}} | {{row.value | number:5}} |
Entropy & Tail Behavior (Qualitative)
Heavy-tailed distributions (e.g., Student t with small $\nu$) allocate more mass to large deviations. In optimization, this can mirror scenarios where gradient noise occasionally spikes—impacting learning rate schedules and adaptive optimizers.
LLM Relevance
- Normal: Initialization, assumed latent noise models.
- Exponential: Token gaps & simplistic arrival processes of rare constructs.
- Beta: Priors over probabilities (e.g., word occurrence rate parameters in Bayesian smoothing).
- Uniform: Baseline uncertainty before learning begins.
- Student t: Robust modeling of embedding coordinate outliers.