← Back to Contents Next Topic →

8.4 Model Evaluation: Testing and Cross-Validation

Introduction

Proper evaluation ensures models generalize beyond training data. This section covers methods for assessing model performance, from simple train/test splits to sophisticated cross-validation strategies. For LLMs, evaluation determines deployment readiness.

1. Train/Test Split

Simplest evaluation method. Divide data into two sets:

  • Training Set (70-80%): Fit model parameters
  • Test Set (20-30%): Evaluate performance

Critical Rule: Test set must remain completely unseen during training. Never use test data to make any decisions about the model.

Interactive: Train/Test Split

Visualize how train/test split affects evaluation:

Train Set

{{splitMetrics.train}}

samples

Test Set

{{splitMetrics.test}}

samples

Test Error

{{splitMetrics.testError | number:3}}

Generalization estimate

2. K-Fold Cross-Validation

More robust than single split. Procedure:

  1. Split data into \( k \) equal folds
  2. For each fold \( i = 1, \ldots, k \):
    • Train on all folds except \( i \)
    • Validate on fold \( i \)
  3. Average performance across all \( k \) folds

Average error: $$\text{CV Error} = \frac{1}{k}\sum_{i=1}^{k}\text{Error}_i$$

Common choices: \( k = 5 \) or \( k = 10 \). Special case: \( k = n \) (Leave-One-Out CV).

Interactive: K-Fold Cross-Validation

See how K-fold CV evaluates model performance:

Fold {{currentFold}} Error

{{kfoldMetrics.currentError | number:3}}

Average CV Error

{{kfoldMetrics.avgError | number:3}}

Std Deviation

{{kfoldMetrics.stdError | number:3}}

2.1 Cross-Validation Strategy Comparison

Different evaluation strategies trade off computational cost, bias, variance, and stability. Compare them below.

Interactive: CV Strategies

{{m.name}}

{{m.total | number:2}}

Composite Score (lower better)

Interpretation: Holdout is fast but high variance; K-Fold balances; LOOCV lowest bias but costly; Stratified reduces variance for imbalanced data; Time Series preserves order.

3. Stratified K-Fold

For classification with imbalanced classes, maintain class distribution in each fold: $$\frac{\text{# samples of class } c \text{ in fold } i}{\text{Total samples in fold } i} \approx \frac{\text{Total samples of class } c}{\text{Total samples}}$$

Ensures each fold is representative of overall class balance.

Interactive: Stratified vs Regular K-Fold

Compare stratified and regular splitting:

{{splitType === 'stratified' ? 'Stratified' : 'Regular'}}: {{stratifiedInfo}}

4. Regression Metrics

R-Squared (R²)

Proportion of variance explained by model: $$R^2 = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}} = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2}$$ Range: \( (-\infty, 1] \). Perfect fit: \( R^2 = 1 \). Worse than mean: \( R^2 < 0 \).

Adjusted R²

Penalizes adding features that don't improve fit: $$R^2_{\text{adj}} = 1 - \frac{(1-R^2)(n-1)}{n-p-1}$$ where \( n \) is sample size, \( p \) is number of features.

Root Mean Squared Error (RMSE)

Square root of MSE, in same units as target: $$\text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2}$$

Interactive: Regression Metrics

See how different fits affect metrics:

R²

{{regMetrics.r2 | number:3}}

RMSE

{{regMetrics.rmse | number:2}}

MAE

{{regMetrics.mae | number:2}}

5. Classification Metrics

Confusion Matrix

For binary classification:

Predicted
Positive Negative
Actual Positive True Positive (TP) False Negative (FN)
Negative False Positive (FP) True Negative (TN)

Accuracy

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$ Misleading for imbalanced datasets.

Precision

Of predicted positives, how many are correct: $$\text{Precision} = \frac{TP}{TP + FP}$$

Recall (Sensitivity)

Of actual positives, how many found: $$\text{Recall} = \frac{TP}{TP + FN}$$

F1 Score

Harmonic mean of precision and recall: $$F_1 = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

Interactive: Classification Metrics

Adjust threshold to see metric tradeoffs:

Accuracy

{{classMetrics.accuracy | number:3}}

Precision

{{classMetrics.precision | number:3}}

Recall

{{classMetrics.recall | number:3}}

F1 Score

{{classMetrics.f1 | number:3}}

5.1 Precision-Recall Curve

Precision-Recall (PR) curves focus on performance for the positive class and are especially informative for imbalanced datasets where ROC curves can appear overly optimistic.

Interactive: Precision-Recall Curve

AP (Approx)

{{prMetrics.ap | number:3}}

Average Precision

Best F1

{{prMetrics.bestF1 | number:3}}

Max across thresholds

Threshold*

{{prMetrics.bestThresh | number:2}}

For best F1

Note: AP summarizes area under the PR curve (approximation). F1 threshold shows operating point maximizing harmonic mean of precision and recall.

6. ROC Curve and AUC

Receiver Operating Characteristic (ROC) curve plots True Positive Rate vs False Positive Rate at different classification thresholds: $$\text{TPR} = \frac{TP}{TP + FN}, \quad \text{FPR} = \frac{FP}{FP + TN}$$

Area Under Curve (AUC): Probability that model ranks random positive example higher than random negative example. Perfect classifier: AUC = 1. Random: AUC = 0.5.

Interactive: ROC Curve

See how model quality affects ROC curve:

AUC: {{rocMetrics.auc | number:3}} - {{rocMetrics.interpretation}}

7. Time Series Cross-Validation

For sequential data (like text), use forward-chaining:

  1. Split 1: Train on [1:n], test on [n+1:2n]
  2. Split 2: Train on [1:2n], test on [2n+1:3n]
  3. Split 3: Train on [1:3n], test on [3n+1:4n]
  4. And so on...

Respects temporal order, prevents look-ahead bias.

Interactive: Time Series CV

Visualize forward-chaining cross-validation:

Split {{currentTSSplit}}: Training on first {{currentTSSplit * 20}}%, testing on next 20%

8. LLM Evaluation Specifics

Common LLM Metrics

  • Perplexity: How well model predicts next token
  • BLEU Score: For translation/generation (n-gram overlap)
  • ROUGE Score: For summarization (recall-oriented)
  • Exact Match: For QA tasks (answer exactly correct)
  • F1: For entity extraction, token-level classification

Evaluation Best Practices

  • Use separate test set from different time period or source
  • Evaluate on diverse, representative examples
  • Consider multiple metrics (not just accuracy)
  • Test edge cases and adversarial examples
  • Human evaluation for generation quality

Key Takeaways

  • Train/test split provides basic performance estimate
  • K-fold cross-validation gives robust average performance
  • Stratified CV maintains class balance in each fold
  • Use R² and RMSE for regression, precision/recall/F1 for classification
  • ROC/AUC evaluates classifier across all thresholds
  • Time series CV respects temporal order
  • LLMs need task-specific metrics beyond accuracy
← Back to Contents Next Topic →