9.3 Regularized Linear Regression (Regression II)

Introduction

Regularization adds penalty term to loss function to control model complexity. Prevents overfitting by shrinking coefficients toward zero. Essential technique for high-dimensional data and when features are correlated.

1. Why Regularization?

Problems with OLS

Overfitting: With many features, model fits training noise
Multicollinearity: Correlated predictors lead to unstable estimates
High variance: Small data changes cause large coefficient changes
Poor generalization: Great training performance, poor test performance

Regularization Solution

Add penalty that discourages large coefficients. Trade off fit quality for simpler, more stable model. Implements bias-variance tradeoff explicitly.

Interactive: OLS vs Regularized

Compare OLS and regularized regression:

Number of Features: {{numFeatures}} Sample Size: {{sampleSize}}

OLS Test Error

Regularized Test Error

Improvement

{{compMetrics.improvement | number:1}}%

2. Regularized Objective Function

General form: $$\min_{\boldsymbol{\beta}} \underbrace{||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||^2}_{\text{Data Fit}} + \underbrace{\lambda \cdot \text{Penalty}(\boldsymbol{\beta})}_{\text{Regularization}}$$

where:

First term: Measures fit to training data (sum of squared residuals)
Second term: Penalty for model complexity
$\lambda \geq 0$: Regularization strength (hyperparameter)

Role of λ

$\lambda = 0$: No regularization (standard OLS)
$\lambda$ small: Slight penalty, close to OLS
$\lambda$ large: Strong penalty, coefficients shrink toward zero
$\lambda \to \infty$: All coefficients become zero (intercept-only model)

Interactive: Effect of λ

See how regularization strength affects coefficients:

λ (log scale): 10^{{lambdaSlider}} = {{lambda | number:3}}

Observation: As λ increases, coefficients shrink toward zero

Number of non-zero coefficients: {{lambdaMetrics.nonZero}}/{{lambdaMetrics.total}}

Regularization Paths

Explore how each coefficient changes as λ varies (log scale). Lasso creates sparsity by setting coefficients to zero; Ridge shrinks all coefficients smoothly.

Path Type: Features: {{regPath.p}} Samples: {{regPath.n}} Noise: {{regPath.noise}}

Final Sparsity

Zero Coefs

||β||₁ (final)

||β||₂ (final)

λ range

10^{-3}→10^{2}

Insight: Lasso selects features by driving coefficients to zero; Ridge retains all features.

3. Types of Regularization

L2 Regularization (Ridge)

Penalty is sum of squared coefficients: $$\text{Penalty} = ||\boldsymbol{\beta}||_2^2 = \sum_{j=1}^{p}\beta_j^2$$ Objective: $$\min_{\boldsymbol{\beta}} ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||^2 + \lambda \sum_{j=1}^{p}\beta_j^2$$

Properties:

Shrinks coefficients smoothly toward zero
Never sets coefficients exactly to zero (all features retained)
Closed-form solution exists
Works well with correlated predictors

L1 Regularization (Lasso)

Penalty is sum of absolute values: $$\text{Penalty} = ||\boldsymbol{\beta}||_1 = \sum_{j=1}^{p}|\beta_j|$$ Objective: $$\min_{\boldsymbol{\beta}} ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||^2 + \lambda \sum_{j=1}^{p}|\beta_j|$$

Properties:

Can set coefficients exactly to zero (feature selection)
No closed-form solution (requires iterative methods)
Produces sparse models
Struggles when features highly correlated (picks one arbitrarily)

Elastic Net

Combines L1 and L2: $$\text{Penalty} = \alpha ||\boldsymbol{\beta}||_1 + (1-\alpha)||\boldsymbol{\beta}||_2^2$$

Balances benefits of both Ridge and Lasso.

Interactive: L1 vs L2 Penalty Shapes

Visualize constraint regions in 2D coefficient space:

Penalty Type: α (L1 weight): {{alpha}}

Key insight: L1's corners encourage exact zeros; L2's smooth circle doesn't

4. Bias-Variance Tradeoff

Regularization explicitly manages bias-variance tradeoff:

λ = 0 (OLS): Low bias, high variance (overfitting risk)
λ small: Slight increase in bias, variance reduction
λ optimal: Best balance - minimizes test error
λ large: High bias, low variance (underfitting)

Interactive: Bias-Variance Curve

See how test error varies with λ:

Data Noise: {{biasVarNoise}}/10

Optimal λ: {{biasVarMetrics.optLambda | number:3}}

Minimum test error: {{biasVarMetrics.minError | number:2}}

5. Choosing λ: Cross-Validation

Use k-fold cross-validation to select optimal $\lambda$:

Try range of $\lambda$ values (e.g., $10^{-3}, 10^{-2}, \ldots, 10^2$)
For each $\lambda$, perform k-fold CV
Calculate average validation error across folds
Choose $\lambda$ with lowest CV error
Refit on full training set with chosen $\lambda$

One Standard Error Rule

Choose most regularized model (largest $\lambda$) within one standard error of minimum CV error. Favors simpler models when performance is similar.

Interactive: Cross-Validation for λ

See CV error across different λ values:

Number of Folds: {{numFolds}}

Best λ: {{cvMetrics.bestLambda | number:3}}

CV Error: {{cvMetrics.bestError | number:2}}

1-SE Rule λ: {{cvMetrics.oneSE | number:3}}

6. Standardization

Important: Standardize features before regularization!

Penalty depends on coefficient magnitudes, but magnitudes depend on feature scales. Standardize to mean 0 and standard deviation 1: $$x_{ij}^{\text{std}} = \frac{x_{ij} - \bar{x}_j}{s_j}$$

After fitting, transform coefficients back to original scale for interpretation.

Interactive: Impact of Standardization

See why standardization matters:

Feature 2 Scale: {{scaleFeature}}x Standardize

Without standardization: Large-scale features penalized more

With standardization: Fair penalty across all features

7. Geometric Interpretation

Regularization as constrained optimization: $$\min_{\boldsymbol{\beta}} ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||^2 \quad \text{subject to} \quad ||\boldsymbol{\beta}||_q \leq t$$

$q = 2$: Ridge (circular/spherical constraint)
$q = 1$: Lasso (diamond/hypercube constraint with corners)

Solution is where loss contours touch constraint region. For Lasso, corners promote sparsity.

8. Application to LLMs

Weight decay: L2 regularization standard in neural network training
Embedding matrices: Regularization prevents overfitting to training vocabulary
Attention weights: Dropout acts as regularization in transformers
Fine-tuning: Strong regularization keeps model close to pretrained weights
Sparse models: L1 regularization for pruning less important parameters
Layer normalization: Implicit regularization by constraining activation scale

Interactive: Weight Decay in Training

Simulate effect of weight decay on training:

Weight Decay: {{weightDecay}}

Training Loss: {{decayMetrics.trainLoss | number:3}}

Validation Loss: {{decayMetrics.valLoss | number:3}}

Weight Magnitude: {{decayMetrics.weightNorm | number:2}}

Key Takeaways

Regularization adds penalty to control model complexity
Prevents overfitting by shrinking coefficients
λ controls tradeoff between fit and simplicity
Ridge (L2): Smooth shrinkage, keeps all features
Lasso (L1): Feature selection through exact zeros
Cross-validation selects optimal λ
Always standardize features before regularization
Essential for high-dimensional data and LLM training