2.4 Moments and Measures of Spread

Moments summarize the shape of a distribution beyond simple location. For Large Language Models (LLMs), understanding moments of activation and gradient distributions helps design stable architectures and normalization schemes.

Raw and Central Moments

The k-th raw moment of a random variable $X$ is $\mu'_k = \mathbb{E}[X^k]$. The k-th central moment is $\mu_k = \mathbb{E}[(X-\mathbb{E}[X])^k]$. We have $\mu_1=0$, $\mu_2=\mathrm{Var}(X)$.

Examples: $\mu_3$ captures skewness direction (sign) and magnitude; $\mu_4$ relates to tail weight (kurtosis). High $\mu_4$ in hidden activations can mean more outliers, stressing attention scaling.

Moment Generating Function (MGF)

The MGF (if it exists near 0) is $M_X(t)=\mathbb{E}[e^{tX}]$. Its derivatives at 0 produce raw moments: $M_X^{(k)}(0)=\mu'_k$. For a Normal $(\mu,\sigma^2)$, $M_X(t)=\exp(\mu t + \tfrac{1}{2}\sigma^2 t^2)$. MGFs justify how sums of independent normals stay normal, reflecting linear composition of hidden layers.

Covariance and Correlation

Covariance: $\mathrm{Cov}(X,Y)=\mathbb{E}[(X-\mu_X)(Y-\mu_Y)]$. Correlation (Pearson): $\rho_{XY}=\dfrac{\mathrm{Cov}(X,Y)}{\sigma_X\sigma_Y}$ (when $\sigma_X,\sigma_Y>0$). In multi-head attention, correlation between head outputs influences redundancy and expressivity.

Law of Large Numbers (LLN)

The sample mean $\bar{X}_n=\frac{1}{n}\sum_{i=1}^n X_i$ converges to $\mathbb{E}[X]$ as $n\to\infty$. We empirically illustrate this for different distributions.

Sampling Controls

Distribution:

μ: {{params.normal.mu | number:2}}
σ: {{params.normal.sigma | number:2}}

λ: {{params.exponential.lambda | number:2}}

a: {{params.uniform.a | number:2}}
b: {{params.uniform.b | number:2}}

Samples per Tick: {{batchSize}}

Total Samples n: {{n}}

Sample Mean: {{sampleMean | number:4}}

Theoretical Mean: {{theoretical.mean | number:4}}

Absolute Error: {{Math.abs(sampleMean - theoretical.mean) | number:4}}

Covariance Explorer

Generate pairs $(X,Y)$ with controllable correlation via: $Y = a X + b Z$ where $Z \sim \text{Normal}(0,1)$ independent.

a: {{covParams.a | number:2}}
b: {{covParams.b | number:2}}

Estimated Corr: {{corr | number:4}}

MGF Existence Caveat

Not every distribution has an MGF (e.g., Cauchy). For heavy-tailed gradients, relying only on variance can be misleading—robust metrics (median absolute deviation) or characteristic functions may be preferable.

LLM Connection

Raw Moments: Early-layer embedding coordinate scales influence downstream normalization.
Covariance: High inter-head correlation reduces effective model capacity; decorrelated representations aid diversity.
LLN: Gradient accumulation over micro-batches approximates expectation, stabilizing updates.
Higher Moments: Monitoring kurtosis can guide gradient clipping thresholds.