8.2 Bias-Variance Tradeoff
Introduction
The bias-variance tradeoff is fundamental to understanding model performance. All prediction errors come from three sources: bias, variance, and irreducible noise. Understanding this decomposition helps choose appropriate model complexity.
1. Error Decomposition
For a model \(\hat{f}(x)\) predicting target \(y\), the expected prediction error at a point \(x\) is: $$\mathbb{E}\left[(y - \hat{f}(x))^2\right] = \text{Bias}^2 + \text{Variance} + \sigma^2$$
where:
- Bias: \(\text{Bias}(\hat{f}(x)) = \mathbb{E}[\hat{f}(x)] - f(x)\)
- Variance: \(\text{Var}(\hat{f}(x)) = \mathbb{E}[(\hat{f}(x) - \mathbb{E}[\hat{f}(x)])^2]\)
- Irreducible Error: \(\sigma^2\) (noise in data itself)
2. Understanding Bias
Bias measures how far the average prediction is from the true value. High bias means the model makes systematic errors, failing to capture the underlying pattern.
Causes of High Bias:
- Model too simple (underfitting)
- Wrong assumptions about data (e.g., assuming linear when non-linear)
- Insufficient features
Example: Fitting a straight line to parabolic data.
3. Understanding Variance
Variance measures how much predictions change if we train on different datasets. High variance means model is too sensitive to training data fluctuations.
Causes of High Variance:
- Model too complex (overfitting)
- Too many parameters relative to data
- Noise in training data heavily influences model
Example: High-degree polynomial fitting random noise.
Interactive: Bias and Variance Visualization
See how bias and variance affect predictions:
Bias²
{{bvMetrics.bias | number:3}}
{{biasLevel}}
Variance
{{bvMetrics.variance | number:3}}
{{varianceLevel}}
Total Error
{{bvMetrics.total | number:3}}
Bias² + Var + Noise
4. The Tradeoff
As model complexity increases:
- Bias decreases: Model can fit more complex patterns
- Variance increases: Model becomes more sensitive to training data
The optimal model complexity minimizes total error: $$\text{Total Error} = \text{Bias}^2 + \text{Variance} + \sigma^2$$
Interactive: Bias-Variance Tradeoff Curve
See how error components change with model complexity:
Optimal Complexity: ~{{optimalComplexity}} (where total error is minimized)
Key Insight: Simple models have high bias, complex models have high variance
4.1 Bias-Variance Decomposition Calculator
Interactively compute bias², variance, noise, and total expected error for a hypothetical model.
Interactive: Decomposition Calculator
Bias²
{{calc.bias2 | number:4}}
Systematic error
Variance
{{calc.variance | number:4}}
Data sensitivity
Noise (σ²)
{{calc.noiseVar | number:4}}
Irreducible
Total Error
{{calc.total | number:4}}
Bias² + Var + σ²
Status
{{calc.status}}
Diagnostic
Interpretation: Decreasing bias (↑ complexity, epochs) can raise variance. Seek lowest stable total error.
4.2 Bias-Variance Tradeoff 3D Surface
A pseudo-3D heatmap projection of error components across model complexity and noise levels. Rotate the view & toggle component.
Interactive: Error Surface
Reading the Surface: Cooler colors = lower error. Bias² shrinks with complexity; variance grows; noise independent of complexity. Total error shows ridge valley (optimal region).
5. Practical Manifestations
High Bias (Underfitting)
| Symptoms | High training error, high validation error |
| Causes | Too simple model, insufficient features |
| Solutions | Add features, increase model complexity, reduce regularization |
High Variance (Overfitting)
| Symptoms | Low training error, high validation error (large gap) |
| Causes | Too complex model, insufficient data |
| Solutions | Get more data, reduce complexity, add regularization, dropout |
Interactive: Underfitting vs Overfitting
Compare different fitting scenarios:
Train Error: {{fittingMetrics.trainError | number:3}} | Val Error: {{fittingMetrics.valError | number:3}}
Status: {{fittingStatus}}
6. Bias-Variance in Different Models
Linear Regression
- Bias: High if true relationship is non-linear
- Variance: Low (stable predictions)
Decision Trees (Deep)
- Bias: Low (can fit complex patterns)
- Variance: High (sensitive to data changes)
Neural Networks
- Bias: Can be very low with sufficient capacity
- Variance: Can be very high without regularization
Ensemble Methods (Random Forest)
- Bias: Similar to individual trees
- Variance: Reduced through averaging
Interactive: Model Comparison
Compare bias-variance characteristics of different models:
7. Managing the Tradeoff
Reduce Bias
- Increase model complexity
- Add more features or polynomial terms
- Use more flexible models (neural networks, trees)
- Reduce regularization strength
Reduce Variance
- Get more training data
- Reduce model complexity
- Add regularization (L1, L2, dropout)
- Use ensemble methods (bagging, boosting)
- Early stopping
- Data augmentation
Interactive: Regularization Effect
See how regularization controls bias-variance tradeoff:
Training Error
{{regMetrics.train | number:3}}
Validation Error
{{regMetrics.val | number:3}}
Complexity Penalty
{{regMetrics.penalty | number:2}}
8. LLM Context
Pre-training vs Fine-tuning
- Large Pre-trained Models: Low bias (can represent complex patterns)
- Risk: High variance if fine-tuning dataset is small
- Solution: Freeze most layers, only fine-tune top layers
Model Size Considerations
- Small Models: May have high bias on complex language tasks
- Large Models: Low bias but require massive data to avoid variance
- Optimal: Match model capacity to data availability and task complexity
Regularization in LLMs
- Dropout (randomly drop neurons during training)
- Weight decay (L2 regularization on parameters)
- Early stopping (stop when validation loss increases)
- Data augmentation (back-translation, paraphrasing)
Interactive: LLM Training Dynamics
Simulate training a language model:
Analysis: {{llmAnalysis}}
Recommendation: {{llmRecommendation}}
Key Takeaways
- All prediction error splits into bias², variance, and irreducible noise
- Bias measures systematic errors; variance measures sensitivity to data
- Simple models → high bias; complex models → high variance
- Optimal model balances both to minimize total error
- Use learning curves and cross-validation to diagnose bias vs variance
- For LLMs: balance model size with available training data