3.3 Distances Between Distributions

Comparing probability distributions is fundamental for model evaluation, optimization, and aligning outputs with targets. Different metrics capture different geometric or information-theoretic aspects. In LLM fine-tuning and distillation, choosing the right divergence affects gradient signals and stability.

Total Variation (TV): $\mathrm{TV}(P,Q) = \tfrac{1}{2}\int |p(x)-q(x)|\,dx$.

L₁ Distance: $\|P-Q\|_1 = \int |p-q|$ (so $\mathrm{TV}=\tfrac12\|P-Q\|_1$).

L₂ Distance: $\|P-Q\|_2 = \left( \int (p-q)^2 dx \right)^{1/2}$.

Jensen–Shannon Divergence: $\mathrm{JSD}(P,Q) = \tfrac{1}{2} D_{KL}(P\|M)+\tfrac{1}{2}D_{KL}(Q\|M)$ with $M=\tfrac{1}{2}(P+Q)$.

1-Wasserstein: $W_1(P,Q)= \inf_{\gamma\in\Pi(P,Q)} \int |x-y| d\gamma(x,y)$ (earth mover's distance).

Two Gaussian Distributions

Adjust parameters of two univariate Gaussians $P=\mathcal{N}(\mu_1,\sigma_1^2)$ and $Q=\mathcal{N}(\mu_2,\sigma_2^2)$. We approximate distances numerically on a finite grid. For Gaussians, some closed forms exist (e.g., $W_2$), but we show generic numerical integration to mimic how token logits are compared in practice (softmax probabilities discretized over vocabulary).

Parameters

μ₁: {{p1.mu | number:2}}

σ₁: {{p1.sigma | number:2}}

μ₂: {{p2.mu | number:2}}

σ₂: {{p2.sigma | number:2}}

Grid Width: {{gridWidth}}

Resolution (N): {{N}}

Domain integrates on [c - gridWidth, c + gridWidth] where c=(μ₁+μ₂)/2.

L1 (∫|p−q|): {{metrics.L1 | number:5}}

TV (0.5 L1): {{metrics.TV | number:5}}

L2: {{metrics.L2 | number:5}}

JSD: {{metrics.JSD | number:5}}

W1 (approx): {{metrics.W1 | number:5}}

Overlap Mass: {{metrics.overlap | number:5}}

Overlap ≈ ∫ min(p,q) dx. Relates to classification Bayes error lower bound 0.5(1 - TV).

PDFs and Overlap

Purple shaded region = pointwise min(p,q). Dashed green line spans μ₁ → μ₂ (contributing to Wasserstein mass displacement).

Metric Intuitions

L1 / TV: Direct discrepancy in probability mass. TV bounds difference in expectations of bounded functions.
L2: Emphasizes peaks; sensitive to squared differences where one distribution is sharp.
JSD: Symmetric, always finite, relates to average information gain distinguishing $P$ vs $Q$. Square root of JSD is a metric.
Wasserstein: Reflects geometric cost of transporting mass. Stable under support mismatches; vital for diffusion & generative alignment.
Overlap Mass: Higher overlap implies easier knowledge distillation (student mimics teacher with fewer corrective gradients).

LLM Relevance

KL vs JSD: KL is asymmetric and can explode if support mismatch; JSD is safer for comparing model vs reference distributions.
Alignment: Reinforcement Learning from Human Feedback sometimes monitors distribution shifts with TV or Wasserstein-like penalties.
Distillation: Matching softened teacher logits approximates minimizing KL which relates to JSD and overlap improvements.
Mode Collapse: In generative models, Wasserstein distances penalize collapsed support more smoothly than KL.