7.2 Data Visualization: Histogram
Understanding Histograms in Machine Learning
Histograms are fundamental tools for understanding data distributions, essential for building robust LLMs and machine learning models. They reveal patterns, outliers, and distributional properties that guide model selection and feature engineering.
Mathematical Foundation
A histogram divides the range of data values into bins and counts the frequency of observations in each bin. For a dataset X = {x₁, x₂, ..., xₙ}, the histogram function h(x) is defined as:
Where bᵢ represents the i-th bin boundary and n is the total number of observations.
Interactive Histogram Generator
{{sampleSize}}
Bin Width Selection: The Bias-Variance Tradeoff
Choosing optimal bin width involves balancing bias and variance:
- Too few bins (large width): High bias, low variance - smooth but potentially inaccurate
- Too many bins (small width): Low bias, high variance - detailed but noisy
Optimal Bin Width Formulas
Sturges' Rule: k = ⌈log₂(n) + 1⌉
Scott's Rule: h = 3.5σn⁻¹/³
Freedman-Diaconis Rule: h = 2 × IQR × n⁻¹/³
Current Dataset Statistics:
Sample Size (n): {{sampleSize}}
Sturges' Bins: {{sturgesBins}}
Scott's Bins: {{scottBins}}
FD Bins: {{fdBins}}
Current Bins: {{binCount}}
Histogram Properties for LLM Training Data
Skewness
Measures asymmetry: γ₁ = E[(X-μ)³]/σ³
Current: {{statistics.skewness | number:3}}
Kurtosis
Measures tail heaviness: γ₂ = E[(X-μ)⁴]/σ⁴ - 3
Current: {{statistics.kurtosis | number:3}}
Entropy
Information content: H = -Σpᵢlog₂(pᵢ)
Current: {{statistics.entropy | number:3}} bits
Applications in Language Model Development
1. Token Frequency Analysis
Histograms reveal token distribution patterns in training corpora, helping identify:
- Rare tokens that may need special handling
- Common tokens requiring efficient encoding
- Distribution imbalances affecting model performance
2. Attention Weight Distributions
Analyzing attention mechanisms through histograms helps understand:
- Attention concentration patterns
- Model focus distribution across tokens
- Potential attention collapse issues
3. Gradient Distribution Monitoring
Tracking gradient histograms during training reveals:
- Vanishing gradient problems
- Exploding gradient issues
- Optimal learning rate regions
Advanced Histogram Techniques
Adaptive Binning
For non-uniform data distributions, adaptive binning adjusts bin widths based on local density:
Where f(xᵢ) is the local density estimate at point xᵢ.
Cumulative Histograms
The cumulative distribution function (CDF) provides complementary insights:
Practical Example: Text Length Distribution
Consider analyzing sentence lengths in a training corpus:
This histogram shows typical sentence length patterns that inform:
- Maximum sequence length for model architecture
- Padding strategies for batch processing
- Memory allocation requirements
NEW: Distribution Fitting & Goodness-of-Fit
Fit several candidate parametric distributions to the current dataset and compare parameter estimates and KS statistics (lower is better).
Overlay Fits
Q-Q Plot (Selected)
ECDF vs Theoretical CDF
Fit Summary
| Dist | Params (MLE) | KS Stat |
|---|---|---|
| {{f.name}} | {{f.paramText}} | {{f.ks | number:4}} |
NEW: Interactive Hypothesis Testing
Experiment with common significance tests. Modify assumptions and observe p-value changes.