7.3 Kernel Density Estimation (KDE)

Introduction

Kernel Density Estimation is a non-parametric method to estimate the probability density function of a random variable. Unlike histograms, KDE produces smooth, continuous estimates. This is crucial for understanding token embeddings, activation distributions, and attention weight patterns in LLMs.

1. The KDE Formula

Given \( n \) data points \( x_1, x_2, \ldots, x_n \), the kernel density estimate at point \( x \) is: $$\hat{f}(x) = \frac{1}{nh}\sum_{i=1}^{n}K\left(\frac{x - x_i}{h}\right)$$ where:

• \( K(\cdot) \) is the kernel function (satisfies \( \int K(u)du = 1 \))
• \( h \) is the bandwidth (controls smoothness)
• \( n \) is the number of data points

2. Common Kernel Functions

Gaussian Kernel (Most Popular)

$$K(u) = \frac{1}{\sqrt{2\pi}}e^{-\frac{u^2}{2}}$$ Smooth, differentiable, infinite support.

Epanechnikov Kernel (Optimal)

$$K(u) = \frac{3}{4}(1 - u^2) \text{ if } |u| \leq 1, \text{ else } 0$$ Minimizes mean integrated squared error, compact support.

Uniform (Box) Kernel

$$K(u) = \frac{1}{2} \text{ if } |u| \leq 1, \text{ else } 0$$ Simple but not smooth.

3. Bandwidth Selection

The bandwidth \( h \) is the most critical parameter. Too small → undersmoothing (noisy), too large → oversmoothing (loss of detail).

Silverman's Rule of Thumb

$$h = 0.9 \min(\sigma, \text{IQR}/1.34) \cdot n^{-1/5}$$ where \( \sigma \) is standard deviation, IQR is interquartile range, \( n \) is sample size.

Scott's Rule

$$h = \sigma \cdot n^{-1/(d+4)}$$ where \( d \) is the number of dimensions (1 for univariate).

4. KDE vs Histogram

Histograms depend heavily on bin placement and width. KDE provides a smooth, continuous alternative:

• Histogram: Discrete, bin-dependent, simple
• KDE: Continuous, smooth, more informative

5. Multivariate KDE

For \( d \)-dimensional data, the KDE becomes: $$\hat{f}(\mathbf{x}) = \frac{1}{n}\sum_{i=1}^{n}\frac{1}{|\mathbf{H}|^{1/2}}K\left(\mathbf{H}^{-1/2}(\mathbf{x} - \mathbf{x}_i)\right)$$ where \( \mathbf{H} \) is the bandwidth matrix. For LLMs, this helps visualize 2D embedding spaces.

6. Applications in LLM Development

Activation Distribution Analysis

KDE helps visualize neuron activation distributions. Healthy networks show smooth, well-behaved distributions. Dead neurons (always zero) or saturated neurons (always max) indicate training issues.

Attention Weight Distributions

Analyzing attention weight distributions helps understand if the model focuses broadly or narrowly. KDE reveals multimodality that histograms might miss.

Token Embedding Visualization

After dimensionality reduction (t-SNE, UMAP), KDE shows density patterns in embedding space, helping identify semantic clusters and outliers.

Advantages and Limitations

Aspect	Advantages	Limitations
Smoothness	Continuous, differentiable	May oversmooth sharp features
Parameters	Only bandwidth to tune	Bandwidth selection non-trivial
Computation	Simple formula	O(n) per evaluation point
Boundary	No binning artifacts	Boundary bias (can oversmooth edges)
Interpretation	Intuitive probability density	Not a true probability (needs normalization)