← Back to Contents Next Topic →

7.1 Univariate Statistical Measures

Introduction

Univariate statistics describe a single variable's distribution. These measures are fundamental for understanding data before building LLMs. They help us understand token distributions, embedding magnitudes, and model output characteristics.

1. Measures of Central Tendency

Mean (Arithmetic Average)

The mean is the sum of all values divided by the count: $$\mu = \bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i$$ The mean is sensitive to outliers but optimal for symmetric distributions.

Median

The median is the middle value when data is sorted. For \( n \) values: $$\text{Median} = \begin{cases} x_{(n+1)/2} & \text{if } n \text{ is odd} \\ \frac{x_{n/2} + x_{n/2+1}}{2} & \text{if } n \text{ is even} \end{cases}$$ The median is robust to outliers and better for skewed distributions.

Mode

The mode is the most frequent value. A distribution can be:

  • Unimodal: One peak (most common)
  • Bimodal: Two peaks (suggests mixture of populations)
  • Multimodal: Multiple peaks

Interactive: Central Tendency Explorer

Generate data and see how different measures respond:

Mean

{{stats.mean | number:2}}

Median

{{stats.median | number:2}}

Mode

{{stats.mode | number:2}}

2. Measures of Dispersion

Variance

Variance measures the average squared deviation from the mean: $$\sigma^2 = \frac{1}{n}\sum_{i=1}^{n}(x_i - \mu)^2$$ For sample variance (unbiased estimator), divide by \( n-1 \): $$s^2 = \frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar{x})^2$$

Standard Deviation

Standard deviation is the square root of variance: $$\sigma = \sqrt{\sigma^2}$$ It has the same units as the original data, making it more interpretable than variance.

Range and Interquartile Range (IQR)

Range = Maximum - Minimum (sensitive to outliers) $$\text{IQR} = Q_3 - Q_1$$ where \( Q_1 \) is the 25th percentile and \( Q_3 \) is the 75th percentile.

Interactive: Dispersion Comparison

Compare dispersion measures across different spreads:

Low Variance (σ = 1)

High Variance (σ = {{stdDev}})

Variance

{{dispStats.variance | number:2}}

Std Dev

{{dispStats.std | number:2}}

IQR

{{dispStats.iqr | number:2}}

3. Measures of Shape

Skewness

Skewness measures asymmetry: $$\text{Skewness} = \frac{1}{n}\sum_{i=1}^{n}\left(\frac{x_i - \mu}{\sigma}\right)^3$$

  • Skewness = 0: Symmetric (like normal distribution)
  • Skewness > 0: Right-skewed (long tail on right)
  • Skewness < 0: Left-skewed (long tail on left)

Kurtosis

Kurtosis measures tail heaviness: $$\text{Kurtosis} = \frac{1}{n}\sum_{i=1}^{n}\left(\frac{x_i - \mu}{\sigma}\right)^4 - 3$$ (The -3 makes normal distribution have kurtosis = 0)

  • Kurtosis = 0: Normal (mesokurtic)
  • Kurtosis > 0: Heavy tails (leptokurtic) - more outliers
  • Kurtosis < 0: Light tails (platykurtic) - fewer outliers

Interactive: Shape Analysis

Explore how skewness and kurtosis characterize distributions:

Skewness

{{shapeStats.skewness | number:3}}

{{shapeStats.skewLabel}}

Kurtosis

{{shapeStats.kurtosis | number:3}}

{{shapeStats.kurtLabel}}

4. Application to LLM Development

Token Frequency Distribution

Token frequencies in text corpora typically follow a power-law (Zipf's law): $$f(r) \propto \frac{1}{r^\alpha}$$ where \( r \) is the rank and \( \alpha \approx 1 \). This distribution is highly right-skewed.

Interactive: Token Distribution Simulator

Simulate token frequency distribution like in LLM training data:

Observation: Most tokens are rare, few tokens are very common

Top 10% tokens account for: {{topTokensPercentage | number:1}}% of all occurrences

Implication: Need subword tokenization (BPE, WordPiece) to handle rare tokens efficiently

NEW: Animated Central Tendency Convergence

This live simulation streams data points and updates mean, median, and mode over time so you can see their stability and robustness. Add outliers mid-stream to watch the mean react more strongly.

Mean

{{stream.stats.mean | number:2}}

Volatility: {{stream.vol.mean | number:2}}

Median

{{stream.stats.median | number:2}}

Volatility: {{stream.vol.median | number:2}}

Mode

{{stream.stats.mode | number:2}}

Volatility: {{stream.vol.mode | number:2}}

Volatility is rolling standard deviation of last 30 estimates.

NEW: Variance Decomposition Explorer

This tool splits data into groups to illustrate total variance = within-group variance + between-group variance. Adjust group separation and noise.

Grouped Data

Variance Components

Total Var

{{varDec.stats.total | number:2}}

Between Var

{{varDec.stats.between | number:2}}

{{varDec.stats.betweenPct | number:1}}% of total

Within Var

{{varDec.stats.within | number:2}}

{{varDec.stats.withinPct | number:1}}% of total

Formula: Var(total) = Var_between + Var_within (decomposition of ANOVA for equal group sizes).

NEW: Bootstrap Sampling Distribution

Bootstrap resampling approximates the sampling distribution of a statistic by repeatedly sampling (with replacement) from the observed data. Watch how the distribution of the statistic stabilizes and the estimated standard error converges.

Sampling Distribution (Statistic)

Std Error Convergence

Observed Stat

{{bootstrap.observed | number:3}}

Bootstrap Mean

{{bootstrap.meanStat | number:3}}

Std Error

{{bootstrap.se | number:3}}

95% CI

[{{bootstrap.ci[0] | number:2}}, {{bootstrap.ci[1] | number:2}}]

CI computed via percentile method. Increasing B refines the estimate.

Summary Table

Measure Formula Interpretation Robust to Outliers?
Mean $\frac{1}{n}\sum x_i$ Average value No
Median Middle value Center point Yes
Variance $\frac{1}{n}\sum(x_i-\mu)^2$ Average squared deviation No
Std Dev $\sqrt{\sigma^2}$ Typical deviation from mean No
IQR $Q_3 - Q_1$ Middle 50% spread Yes
Skewness $\frac{1}{n}\sum(\frac{x_i-\mu}{\sigma})^3$ Asymmetry direction No
Kurtosis $\frac{1}{n}\sum(\frac{x_i-\mu}{\sigma})^4-3$ Tail heaviness No
← Back to Contents Next Topic →