3.4 Kullback-Leibler (KL) Divergence

The Kullback–Leibler divergence measures the inefficiency of assuming distribution $Q$ when the true distribution is $P$. It is not a symmetric distance but is central in maximum likelihood, variational inference, and LLM training objectives.

Discrete: $D_{KL}(P\|Q) = \sum_i P_i \log\frac{P_i}{Q_i}$ (define $0\log 0 = 0$; requires $Q_i>0$ when $P_i>0$).

Continuous: $D_{KL}(P\|Q)=\int p(x) \log \frac{p(x)}{q(x)} dx$.

Cross-Entropy: $H(P,Q) = -\sum_i P_i \log Q_i = H(P) + D_{KL}(P\|Q)$.

Entropy: $H(P) = -\sum_i P_i \log P_i$.

Categorical Editor (Vocabulary Toy Example)

Adjust logits for model distribution $Q$ and target probabilities $P$. Logits are converted via softmax. Observe how $D_{KL}(P\|Q)$ and $D_{KL}(Q\|P)$ differ. This mimics next-token training where $P$ may be a smoothed or label-distribution and $Q$ is model output.

Probabilities / Logits

i	Target P_i	Model Logit z_i	Q_i
{{$index}}		{{Z[$index] \| number:1}}	{{Q[$index] \| number:4}}

P auto-normalized. Adjust logits to shape Q.

Temperature τ: {{temp | number:2}}

Information Measures (nat units)

H(P): {{metrics.Hp | number:5}}

H(P,Q): {{metrics.HpQ | number:5}}

D_KL(P||Q): {{metrics.Dpq | number:5}}

D_KL(Q||P): {{metrics.Dqp | number:5}}

JSD(P,Q): {{metrics.JSD | number:5}}

Perplexity exp(H(P,Q)): {{metrics.perp | number:3}}

Convert to bits: divide by ln 2. Perplexity corresponds to effective branching factor.

Continuous Approximation (Gaussians)

For $\mathcal{N}(\mu_1,\sigma_1^2)$ and $\mathcal{N}(\mu_2,\sigma_2^2)$:

$D_{KL}(\mathcal{N}_1\|\mathcal{N}_2)= \log\frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1-\mu_2)^2}{2\sigma_2^2} - \frac{1}{2}$

μ₁: {{g1.mu | number:2}}

σ₁: {{g1.sigma | number:2}}

μ₂: {{g2.mu | number:2}}

σ₂: {{g2.sigma | number:2}}

KL(N1||N2): {{gaussKL12 | number:5}}

KL(N2||N1): {{gaussKL21 | number:5}}

Gradient Intuition

Minimizing $D_{KL}(P\|Q)$ w.r.t. model parameters in $Q$ (when $P$ fixed) is equivalent to minimizing cross-entropy $H(P,Q)$. Gradient of cross-entropy for logits $z_j$ after softmax is $\partial H/\partial z_j = Q_j - P_j$. This forms the core of next-token prediction training.

Asymmetry & Mode-Seeking vs Mode-Covering

$D_{KL}(P\|Q)$ penalizes when $Q$ assigns low probability where $P$ has mass (mode-covering). $D_{KL}(Q\|P)$ penalizes assigning probability where $P$ has none (mode-seeking). This asymmetry matters in variational inference vs reverse KL settings.

LLM Relevance

Training Objective: Standard next-token loss is cross-entropy, i.e., minimizing $D_{KL}(P_{data}\|Q_{model})$ plus constant $H(P_{data})$.
Label Smoothing: Alters $P$ to reduce overconfidence, lowering $D_{KL}$ sensitivity to rare tokens.
Distillation: Teacher $P$ is soft distribution; student matches full token support, reducing KL and improving generalization.
Sampling Temperature: Scales logits: $Q_i(\tau)=\frac{\exp(z_i/\tau)}{\sum_j\exp(z_j/\tau)}$; affects entropy and downstream KL to data distribution.