2.1 Continuous Variables and Distribution

A continuous random variable (RV) can take any real value within an interval. Unlike a discrete RV, we do not assign probability to exact points. Instead, probability lives over intervals.

$P(X = x) = 0 \quad \text{(for a true continuous RV)}, \qquad P(a \le X \le b) = \int_{a}^{b} f_X(x)\,dx$

The function $f_X(x)$ is the probability density function (PDF). It satisfies two axioms:

$f_X(x) \ge 0$ for all $x$.
$\int_{-\infty}^{+\infty} f_X(x)\,dx = 1$ (normalization).

Cumulative Distribution Function (CDF)

The CDF accumulates probability from $-\infty$ up to a point $x$:

$F_X(x) = P(X \le x) = \int_{-\infty}^{x} f_X(t)\,dt$

Properties: $F_X$ is non-decreasing, right-continuous, $\lim_{x\to -\infty} F_X(x)=0$, $\lim_{x\to +\infty} F_X(x)=1$ and $f_X(x) = \dfrac{d}{dx}F_X(x)$ wherever differentiable.

Expectation and Variance

For a continuous RV $X$ with PDF $f_X$:

$\mathbb{E}[X] = \int_{-\infty}^{+\infty} x f_X(x)\,dx, \qquad \mathrm{Var}(X) = \int_{-\infty}^{+\infty} (x-\mu)^2 f_X(x)\,dx = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$

Key Examples

Uniform $(a,b)$: $f(x)= \dfrac{1}{b-a}$ for $a\le x\le b$, else 0. Mean $= \dfrac{a+b}{2}$, variance $= \dfrac{(b-a)^2}{12}$.
Normal $(\mu,\sigma^2)$: $f(x)= \dfrac{1}{\sigma\sqrt{2\pi}} e^{-(x-\mu)^2/(2\sigma^2)}$. Central for LLM weight initialization & noise modeling.
Exponential $(\lambda)$: $f(x)= \lambda e^{-\lambda x}$ for $x\ge 0$. Memoryless; analogous to waiting time until a rare token type appears.

Why Continuous Variables Matter for LLMs

Although token outputs are discrete, internal representations (embeddings, hidden states, attention scores) are continuous vectors. Their distributions influence stability, gradient flow, and information propagation. Initialization schemes attempt to keep variances in controlled ranges to avoid exploding/vanishing activations.

Interactive PDF Explorer (Normal & Uniform)

Adjust parameters to see how the PDF changes. You can also compute the probability mass in an interval $[L, U]$ (area under the curve) using numerical integration (trapezoidal rule) applied to sampled points. This mimics how continuous integrals are approximated in practice.

Parameters

Distribution Type:

Mean (μ): {{params.normal.mu | number:2}} Std Dev (σ): {{params.normal.sigma | number:2}}

a: {{params.uniform.a | number:2}} b: {{params.uniform.b | number:2}}

λ: {{params.exponential.lambda | number:2}}

Interval Probability

L: {{interval.L | number:2}} U: {{interval.U | number:2}}

Ensuring L <= U automatically.

Approx Probability P(L ≤ X ≤ U): {{intervalProb | number:4}}

Mean (theoretical): {{theoretical.mean | number:4}}

Variance (theoretical): {{theoretical.var | number:4}}

Mathematical Forms Used Internally

Normal PDF: $f(x)=\frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}$ | Uniform PDF: $f(x)=\frac{1}{b-a}$.
Exponential PDF: $f(x)=\lambda e^{-\lambda x}\; (x\ge0)$.

Connection to LLM Stability

If hidden activations $h$ are modeled as approximately normal with mean 0 and variance $\sigma^2$, then layer normalization rescales them to control distributional drift over depth. Probability densities help reason about saturation of nonlinearities and gradient magnitudes.

Further Thought

Consider how changing $\sigma$ widens the distribution—this implies more uncertainty in activation magnitude. In contrast, a narrow distribution (small $\sigma$) might reduce expressiveness but increase stability. Many initialization heuristics (Xavier, Kaiming) implicitly target a desired variance propagation profile.