← Back to Contents Next Topic →

2.3 Features of Probability Distribution

A probability distribution is not fully described by the mean and variance alone; higher-order structure and shape details matter for modeling, optimization stability, and generalization in LLMs. Here we explore central tendency, dispersion, shape, and quantiles.

Mean (Expectation)

$\mathbb{E}[X] = \int x f_X(x)\,dx$ (continuous) or $\sum_x x P(X=x)$ (discrete). In embeddings, coordinate means near zero help avoid directional bias.

Variance

$\mathrm{Var}(X) = \mathbb{E}[(X-\mu)^2] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$. Controls scale magnitude; improper variance propagation causes gradient explosion or vanishing.

Skewness

Measures asymmetry: $\gamma_1 = \dfrac{\mathbb{E}[(X-\mu)^3]}{\sigma^3}$. Positive skew: long right tail. In optimization noise, skewness can bias update directions.

Kurtosis (Excess)

Tail heaviness: $\gamma_2 = \dfrac{\mathbb{E}[(X-\mu)^4]}{\sigma^4} - 3$. Normal distribution has excess kurtosis 0. Heavy tails (kurtosis > 0) imply more extreme values—affecting layer norm robustness.

Quantiles & Median

The $p$-quantile $q_p$ solves $F(q_p)=p$. Median $q_{0.5}$. For token probability distributions in LLM decoding, quantiles inform nucleus thresholds.

Empirical Estimation

From i.i.d. samples $x_1,\dots,x_n$:

  • Sample mean: $\bar{x} = \dfrac{1}{n}\sum_i x_i$ (unbiased for mean).
  • Sample variance: $s^2 = \dfrac{1}{n-1}\sum_i (x_i-\bar{x})^2$ (unbiased for variance).
  • Sample skewness (Fisher): $g_1 = \dfrac{n}{(n-1)(n-2)} \sum_i \dfrac{(x_i-\bar{x})^3}{s^3}$.
  • Sample excess kurtosis: $g_2 = \frac{n(n+1)}{(n-1)(n-2)(n-3)} \sum_i \frac{(x_i-\bar{x})^4}{s^4} - \frac{3(n-1)^2}{(n-2)(n-3)}$.

Interactive Simulation

Choose a base distribution, sample size, and observe empirical statistics converging toward theoretical values. This illustrates finite-sample variability crucial to understanding stochastic gradient noise in LLM training.

μ: {{params.normal.mu | number:2}}
σ: {{params.normal.sigma | number:2}}
λ: {{params.exponential.lambda | number:2}}
a: {{params.uniform.a | number:2}}
b: {{params.uniform.b | number:2}}
ν: {{params.student.nu | number:0}}

Empirical Statistics

Mean{{stats.mean | number:4}}Theoretical: {{theoretical.mean | number:4}}
Variance{{stats.var | number:4}}Theoretical: {{theoretical.var | number:4}}
Skewness{{stats.skew | number:4}}Theoretical: {{theoretical.skew | number:4}}
Excess Kurtosis{{stats.kurt | number:4}}Theoretical: {{theoretical.kurt | number:4}}
Median{{stats.median | number:4}}Theoretical: {{theoretical.median | number:4}}
q0.9{{stats.q90 | number:4}}Theoretical: {{theoretical.q90 | number:4}}

LLM Perspective

Monitoring higher moments of activation distributions during training can signal emerging instability (e.g., increasing kurtosis indicating heavy-tailed gradients). Some optimization research explores re-centering or clipping strategies informed by these distributional features.

← Back to Contents Next Topic →