9.4 Ridge and Lasso Regression

Introduction

Ridge and Lasso are two most popular regularization techniques. Both add penalty to prevent overfitting, but differ in type of penalty and resulting behavior. Ridge shrinks all coefficients smoothly toward zero, while Lasso can set some coefficients exactly to zero (automatic feature selection).

1. Ridge Regression (L2 Regularization)

Objective Function

$$\min_{\boldsymbol{\beta}} ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||^2 + \lambda \sum_{j=1}^{p}\beta_j^2$$

Or in matrix form: $$\min_{\boldsymbol{\beta}} ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||^2 + \lambda ||\boldsymbol{\beta}||_2^2$$

Closed-Form Solution

Ridge has analytical solution: $$\hat{\boldsymbol{\beta}}^{\text{ridge}} = (\mathbf{X}^T\mathbf{X} + \lambda \mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$$

Adding \(\lambda \mathbf{I}\) ensures matrix is invertible even when \(\mathbf{X}^T\mathbf{X}\) is singular.

Properties

• Shrinkage: Pulls all coefficients toward zero, but never exactly zero
• All features retained: No automatic feature selection
• Handles multicollinearity: Stable estimates even with correlated predictors
• Computational: Fast - closed-form solution
• Bias introduced: Trades bias for reduced variance

2. Lasso Regression (L1 Regularization)

Objective Function

$$\min_{\boldsymbol{\beta}} ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||^2 + \lambda \sum_{j=1}^{p}|\beta_j|$$

Or in matrix form: $$\min_{\boldsymbol{\beta}} ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||^2 + \lambda ||\boldsymbol{\beta}||_1$$

Solution

No closed-form solution due to absolute value. Use iterative algorithms:

• Coordinate descent: Optimize one coefficient at a time
• LARS (Least Angle Regression): Efficient path algorithm
• Proximal gradient: General optimization framework

Properties

• Sparsity: Sets many coefficients exactly to zero
• Feature selection: Automatically selects important features
• Interpretability: Simpler models with fewer features
• Correlated features: Picks one arbitrarily from correlated group
• Bias introduced: Can be high for important features

3. Ridge vs Lasso Comparison

Unified Ridge vs Lasso Shrinkage Explorer

Use a single λ slider to observe coefficient shrinkage and generalization error trade-offs for both Ridge and Lasso on the same synthetic dataset. Lasso induces sparsity (exact zeros) while Ridge spreads shrinkage across all coefficients.

4. Soft Thresholding (Lasso)

Lasso update for coordinate \(j\) uses soft thresholding operator: $$\hat{\beta}_j = \mathcal{S}_{\lambda}(z_j) = \begin{cases} z_j - \lambda & \text{if } z_j > \lambda \\ 0 & \text{if } |z_j| \leq \lambda \\ z_j + \lambda & \text{if } z_j < -\lambda \end{cases}$$

where \(z_j\) is OLS estimate. This creates exact zeros.

5. Elastic Net

Combines Ridge and Lasso to get best of both: $$\min_{\boldsymbol{\beta}} ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||^2 + \lambda\left[\alpha ||\boldsymbol{\beta}||_1 + (1-\alpha)||\boldsymbol{\beta}||_2^2\right]$$

where \(\alpha \in [0,1]\) controls mix:

• \(\alpha = 0\): Pure Ridge
• \(\alpha = 1\): Pure Lasso
• \(0 < \alpha < 1\): Compromise - sparsity + stability

Benefits

• Feature selection like Lasso
• Handles correlated features like Ridge (selects groups)
• More stable than Lasso alone

6. Choosing Between Ridge and Lasso

Use Ridge When:

• Believe most features are relevant
• Features are highly correlated
• Want smooth coefficient shrinkage
• Fast computation needed (closed-form solution)

Use Lasso When:

• Suspect many features are irrelevant
• Want automatic feature selection
• Need interpretable sparse model
• Have many more features than samples (\(p \gg n\))

Use Elastic Net When:

• Want feature selection + stability
• Have groups of correlated features (select groups)
• Unsure which regularization suits problem

7. Practical Considerations

Hyperparameter Tuning

• Always use cross-validation to select \(\lambda\)
• Try logarithmic grid: \(10^{-3}, 10^{-2}, \ldots, 10^2\)
• For Elastic Net, tune both \(\lambda\) and \(\alpha\)

Computational Complexity

• Ridge: \(O(p^3 + np^2)\) for closed-form solution
• Lasso: \(O(np)\) per iteration, many iterations
• Warm starts: Use solution from one \(\lambda\) to initialize next

8. Application to LLMs

Weight Decay (Ridge)

• Standard in neural network training
• Prevents weights from growing too large
• Improves generalization
• Typically \(\lambda \in [10^{-5}, 10^{-2}]\)

Sparsity (Lasso)

• Model pruning and compression
• Remove unimportant parameters
• Reduce model size for deployment
• Lottery ticket hypothesis

Attention Regularization

• Encourage sparse attention patterns
• Focus on relevant tokens
• Improve interpretability

9. Advanced Topics

Group Lasso

Encourage sparsity at group level: $$\lambda \sum_{g=1}^{G}\sqrt{p_g}||\boldsymbol{\beta}_g||_2$$ Either all coefficients in group are zero or all are non-zero.

Adaptive Lasso

Weight penalty by initial estimates: $$\lambda \sum_{j=1}^{p}w_j|\beta_j|$$ where \(w_j = 1/|\hat{\beta}_j^{\text{OLS}}|^\gamma\). Penalizes weak coefficients more.

Fused Lasso

Encourage similar values for adjacent coefficients: $$\lambda_1 \sum_{j=1}^{p}|\beta_j| + \lambda_2\sum_{j=2}^{p}|\beta_j - \beta_{j-1}|$$ Useful for ordered features (time, space).

Key Takeaways

• Ridge (L2) shrinks all coefficients smoothly - no feature selection
• Lasso (L1) creates sparsity - automatic feature selection
• Ridge handles multicollinearity better than Lasso
• Lasso more interpretable due to sparsity
• Elastic Net combines both - best of both worlds
• Always use cross-validation to choose λ
• Ridge has closed-form solution, Lasso requires iterations
• Weight decay (Ridge) is standard in LLM training
• L1 regularization enables model pruning and compression