3.1 The Central Limit Theorem (CLT)

The Central Limit Theorem (CLT) states that the properly normalized sum of many independent, identically distributed (i.i.d.) random variables with finite variance tends toward a Normal (Gaussian) distribution, regardless of the original variable's distribution.

If $X_1, X_2, \dots, X_n$ are i.i.d. with mean $\mu$ and variance $\sigma^2$, then the standardized sum $$ Z_n = \frac{\sum_{i=1}^n X_i - n\mu}{\sigma \sqrt{n}} \xrightarrow{d} \mathcal{N}(0,1) $$ as $n \to \infty$.

In LLM training, gradients are often approximated across large batches. The CLT justifies treating aggregated noise as approximately Gaussian, underpinning optimizer assumptions about variance scaling and learning rate schedules.

Assumptions (Basic Form)

Independence (approximately true across different samples in a batch).
Identical distribution (same data generating mechanism).
Finite variance (no infinite second moment).

Interactive Simulation

Choose a base distribution and number of summands $n$. We repeatedly sample sums and show: (1) histogram of raw sums, (2) standardized histogram with Normal(0,1) overlay.

Base Distribution:

p: {{params.bernoulli.p | number:2}}

λ: {{params.exponential.lambda | number:2}}

n (terms per sum): {{nTerms}} Number of Sums: {{numSums}}

Mean (empirical): {{emp.mean | number:4}} | Theoretical: {{theoretical.mean | number:4}}

Var (empirical): {{emp.var | number:4}} | Theoretical: {{theoretical.var | number:4}}

Std of Sum (theoretical): $\sqrt{n}\sigma$ = {{Math.sqrt(nTerms)*Math.sqrt(single.var) | number:4}}

Standardization: $Z = (S_n - n\mu)/(\sigma\sqrt{n})$

Kolmogorov–Smirnov style rough distance (empirical vs Normal CDF): {{ksApprox | number:4}}

Heavy Tails Caveat

If the underlying distribution has infinite variance (e.g., Cauchy), the classical CLT does not apply; stable distributions arise instead. In LLM gradient noise, occasional outliers can slow convergence toward Gaussianity, motivating clipping or adaptive variance estimation.

LLM Relevance

Batching: Aggregating token-level losses approximates a Normal error signal.
Learning Rate Scaling: Variance of the mean gradient decreases as $1/n$, aligning with warm-up and scaling rules.
Noise Modeling: Approximate Gaussianity enables variance-based adaptive optimizers (Adam, RMSProp) to behave predictably.