12.2 Model Selection and Inference

Introduction

Model selection addresses the question: given multiple candidate models, which should we choose? This involves balancing model complexity with goodness of fit, and making statistically principled decisions about model parameters and structure.

1. Cross-Validation

Cross-validation estimates a model's generalization performance by repeatedly training on different subsets of data and testing on held-out portions.

1.1 K-Fold Cross-Validation

The most common approach:

1. Partition data into $K$ equally-sized folds
2. For each fold $k = 1, \ldots, K$:
   • Train on $K-1$ folds
   • Validate on fold $k$
   • Record error $E_k$
3. Average errors: $CV_{(K)} = \frac{1}{K}\sum_{k=1}^{K} E_k$

Standard choice: $K = 5$ or $K = 10$. Larger $K$ gives less biased estimates but higher variance.

1.2 Leave-One-Out Cross-Validation (LOOCV)

Special case where $K = n$ (number of samples). For each data point $i$:

$$CV_{(n)} = \frac{1}{n}\sum_{i=1}^{n} \ell(y_i, \hat{f}^{(-i)}(\mathbf{x}_i))$$

where $\hat{f}^{(-i)}$ is trained on all data except point $i$.

Advantage: Nearly unbiased estimate of generalization error

Disadvantage: High computational cost ($n$ model fits), high variance

1.3 Stratified Cross-Validation

For classification, ensure each fold has similar class proportions to the full dataset. Reduces variance in performance estimates, especially for imbalanced datasets.

2. Information Criteria

Information criteria provide analytical approximations to cross-validation, balancing goodness of fit with model complexity.

2.1 Akaike Information Criterion (AIC)

$$AIC = 2k - 2\ln(\hat{L})$$

where:

• $k$ is the number of parameters
• $\hat{L}$ is the maximum likelihood of the model

For least squares with Gaussian errors:

$$AIC = n\ln\left(\frac{RSS}{n}\right) + 2k$$

where $RSS = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2$ is the residual sum of squares.

Interpretation: Lower AIC is better. Penalizes model complexity linearly.

2.2 Bayesian Information Criterion (BIC)

$$BIC = k\ln(n) - 2\ln(\hat{L})$$

For least squares:

$$BIC = n\ln\left(\frac{RSS}{n}\right) + k\ln(n)$$

Key difference from AIC: Penalty term $k\ln(n)$ grows with sample size, leading to stronger preference for simpler models as $n$ increases.

2.3 Comparison

Criterion	Penalty	Philosophy
AIC	$2k$	Minimize prediction error
BIC	$k\ln(n)$	Find true model (Bayesian)
AICc	$2k + \frac{2k(k+1)}{n-k-1}$	AIC corrected for small samples

3. Hypothesis Testing in Model Selection

3.1 Likelihood Ratio Test

Compare nested models $M_0$ (null, simpler) and $M_1$ (alternative, more complex):

$$\Lambda = -2\ln\left(\frac{L(M_0)}{L(M_1)}\right) = 2(\ln L(M_1) - \ln L(M_0))$$

Under the null hypothesis, $\Lambda \sim \chi^2_{df}$ where $df = k_1 - k_0$ is the difference in number of parameters.

3.2 F-Test for Linear Models

For nested linear models:

$$F = \frac{(RSS_0 - RSS_1)/(k_1 - k_0)}{RSS_1/(n - k_1)}$$

Under the null, $F \sim F_{k_1-k_0, n-k_1}$.

3.3 Multiple Testing Correction

When testing multiple hypotheses, adjust significance levels to control family-wise error rate:

• Bonferroni: Use $\alpha/m$ for each of $m$ tests (conservative)
• Holm-Bonferroni: Sequential method, less conservative
• Benjamini-Hochberg: Controls false discovery rate instead

4. Model Averaging

Instead of selecting a single model, combine predictions from multiple models:

$$\hat{y} = \sum_{m=1}^{M} w_m \hat{f}_m(\mathbf{x})$$

where $w_m \geq 0$ and $\sum_{m=1}^{M} w_m = 1$.

4.1 Bayesian Model Averaging (BMA)

Weight models by their posterior probabilities:

$$w_m = P(M_m | \mathcal{D}) = \frac{P(\mathcal{D}|M_m)P(M_m)}{\sum_{j=1}^{M} P(\mathcal{D}|M_j)P(M_j)}$$

4.2 Stacking

Learn optimal weights using cross-validation:

1. Train each model on training data
2. Get predictions on validation data
3. Learn weights by fitting a meta-model

5. Nested Cross-Validation

For unbiased model selection with hyperparameter tuning:

1. Outer loop: Split data into K folds for performance estimation
2. Inner loop: Within each training fold, use cross-validation to select hyperparameters
3. Test selected model on outer fold

This provides an unbiased estimate of the entire model selection pipeline's performance.

6. Bootstrap Methods

Bootstrap resampling provides another approach to estimate model uncertainty:

1. Draw $B$ bootstrap samples by sampling $n$ points with replacement from original data
2. Train model on each bootstrap sample
3. Estimate variance and confidence intervals from the $B$ predictions

Out-of-Bag (OOB) Error

On average, each bootstrap sample contains $\approx 63.2\%$ unique observations. The remaining $36.8\%$ (out-of-bag) can be used for validation:

$$OOB\ Error = \frac{1}{n}\sum_{i=1}^{n} \ell\left(y_i, \frac{1}{|B^{(-i)}|}\sum_{b \in B^{(-i)}} \hat{f}_b(\mathbf{x}_i)\right)$$

where $B^{(-i)}$ is the set of bootstrap samples not containing observation $i$.

7. Validation Set Approach

Simple approach: split data into training, validation, and test sets:

• Training set (60-70%): Fit model parameters
• Validation set (15-20%): Select model/hyperparameters
• Test set (15-20%): Final performance evaluation (use only once!)

Advantage: Computationally efficient, simple to implement

Disadvantage: High variance, wastes data, performance depends on split

8. Time Series Considerations

For temporal data, standard cross-validation violates temporal ordering. Use instead:

Forward Chaining (Time Series Split)

1. Train on [1, ..., t], test on [t+1, ..., t+h]
2. Train on [1, ..., t+h], test on [t+h+1, ..., t+2h]
3. Continue expanding training window

Blocked Cross-Validation

Create folds that respect temporal dependencies, possibly with gaps between train and test.

9. Practical Guidelines

When to Use What?

• Small dataset: LOOCV or repeated K-fold CV
• Large dataset: Simple train/validation/test split or 5-fold CV
• Nested models: Likelihood ratio test or F-test
• Non-nested models: AIC/BIC or cross-validation
• Many candidates: AIC/BIC for speed, CV for accuracy
• Imbalanced classes: Stratified CV
• Time series: Forward chaining or blocked CV

Common Pitfalls

• Using test set multiple times (leads to overfitting)
• Not using nested CV when tuning hyperparameters
• Forgetting to standardize within CV folds (data leakage)
• Ignoring temporal structure in time series
• Over-interpreting small differences in CV scores

10. Connection to LLMs

Model selection principles are crucial for LLMs:

• Perplexity: Standard evaluation metric, related to cross-entropy loss
• Held-out validation: Pre-training uses separate validation sets to monitor convergence
• Early stopping: Form of model selection based on validation performance
• Hyperparameter tuning: Learning rate, batch size, architecture choices selected via validation
• Model averaging: Ensemble of multiple LLMs or checkpoints improves robustness
• Scaling laws: Provide analytical model selection guidance for compute budget allocation
• Few-shot evaluation: Cross-validation analogue for in-context learning
• Test set contamination: Critical issue when training on web-scale data

Key Takeaways

• Cross-validation provides robust estimates of generalization performance
• K-fold CV (K=5 or 10) balances bias, variance, and computational cost
• AIC minimizes prediction error; BIC favors simpler models
• Use nested CV when hyperparameter tuning to avoid overfitting
• Information criteria provide efficient alternatives to CV for large model spaces
• Always reserve a test set for final evaluation
• Model averaging can outperform single model selection
• Respect data structure (time series, hierarchical, etc.) in validation