8.3 Loss Functions and Cost Minimization

Introduction

Loss functions quantify prediction error. They guide model training by providing a scalar measure of how well predictions match true values. Different tasks require different loss functions. For LLMs, choosing the right loss function is crucial for optimal performance.

1. Mean Squared Error (MSE)

Most common loss for regression. Measures average squared difference between predictions and true values: $$\text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2$$

Properties:

Always non-negative, equals zero only for perfect predictions
Differentiable everywhere (smooth for gradient descent)
Heavily penalizes large errors (quadratic growth)
Sensitive to outliers

2. Mean Absolute Error (MAE)

Alternative for regression. Uses absolute difference instead of squared: $$\text{MAE} = \frac{1}{n}\sum_{i=1}^{n}|y_i - \hat{y}_i|$$

Properties:

More robust to outliers than MSE
Linear penalty (errors grow linearly)
Not differentiable at zero (can complicate optimization)
All errors weighted equally

Interactive: MSE vs MAE

Compare how MSE and MAE respond to prediction errors:

Number of Predictions: {{numPreds}} Outlier Intensity: {{outlierIntensity}}/10

MSE

Penalizes outliers heavily

MAE

More robust to outliers

RMSE

Root Mean Squared Error

3. Cross-Entropy Loss

Standard loss for classification. Measures difference between predicted probability distribution and true distribution.

Binary Cross-Entropy

For binary classification (two classes): $$\text{BCE} = -\frac{1}{n}\sum_{i=1}^{n}\left[y_i\log(\hat{y}_i) + (1-y_i)\log(1-\hat{y}_i)\right]$$ where $y_i \in \{0, 1\}$ and $\hat{y}_i \in (0, 1)$ is predicted probability.

Categorical Cross-Entropy

For multi-class classification: $$\text{CCE} = -\frac{1}{n}\sum_{i=1}^{n}\sum_{c=1}^{C}y_{ic}\log(\hat{y}_{ic})$$ where $C$ is number of classes, $y_{ic}$ is one-hot encoded true label.

Interactive: Cross-Entropy Loss

Visualize how cross-entropy penalizes incorrect predictions:

True Class: Predicted Prob (Class A): {{probA}}% Predicted Prob (Class B): {{probB}}%

Cross-Entropy Loss: {{ceMetrics.loss | number:3}}

Interpretation: {{ceMetrics.interpretation}}

4. Huber Loss

Combines MSE and MAE advantages. Quadratic for small errors, linear for large errors: $$L_\delta(y, \hat{y}) = \begin{cases} \frac{1}{2}(y - \hat{y})^2 & \text{if } |y - \hat{y}| \leq \delta \\ \delta|y - \hat{y}| - \frac{1}{2}\delta^2 & \text{otherwise} \end{cases}$$ where $\delta$ is the threshold parameter.

Properties:

Smooth everywhere (differentiable)
Less sensitive to outliers than MSE
Behaves like MSE for small errors (smooth gradient)
Behaves like MAE for large errors (bounded gradient)

Interactive: Huber Loss

See how Huber loss transitions between MSE and MAE:

Delta (δ): {{delta}} Show MSE Show MAE Show Huber

Huber Loss: Quadratic within [-δ, δ], linear beyond

Use Case: Robust regression with outliers

5. Perplexity (for Language Models)

Common metric for language models. Exponential of cross-entropy: $$\text{Perplexity} = \exp\left(-\frac{1}{n}\sum_{i=1}^{n}\log P(w_i|w_{

Interpretation: If perplexity is 50, model is as confused as if choosing uniformly from 50 tokens.

Interactive: Perplexity Visualization

See how prediction confidence affects perplexity:

Model Confidence: {{confidence}}% Vocabulary Size: {{vocabSize}}

Perplexity

Cross-Entropy

Bits per Token

6. Hinge Loss (for SVMs)

Used in support vector machines for classification: $$L(y, \hat{y}) = \max(0, 1 - y\hat{y})$$ where $y \in \{-1, +1\}$ and $\hat{y}$ is predicted score.

Encourages margin maximization. Zero loss when prediction is confident and correct.

7. Loss Function Comparison

Loss Function	Task	Robustness	Smoothness
MSE	Regression	Low (sensitive to outliers)	Very smooth
MAE	Regression	High (robust to outliers)	Not smooth at 0
Huber	Regression	Medium (balanced)	Smooth everywhere
Cross-Entropy	Classification	N/A	Smooth
Hinge	Classification (SVM)	High (margin-based)	Not smooth at 1

Interactive: Loss Function Comparison

Compare different loss functions on same dataset:

Data Pattern:

MSE

MAE

Huber

Best Choice

8. Optimization via Gradient Descent

Models minimize loss using gradient descent. Update rule: $$\theta_{t+1} = \theta_t - \alpha \nabla_\theta L(\theta_t)$$ where $\alpha$ is learning rate, $\nabla_\theta L$ is gradient of loss with respect to parameters.

Interactive: Gradient Descent

Watch how gradient descent minimizes loss:

Learning Rate: {{learningRate / 100}} Show Optimization Path

Current Loss: {{gdMetrics.currentLoss | number:3}}

Iterations: {{gdMetrics.iterations}}

8.1 Cost Function Landscape (Linear Regression)

Visualize the MSE surface for a simple linear regression model y = w1 * x + w0 across a grid of (w0, w1) values. Observe the convex bowl shape and an optimization path.

Interactive: Cost Surface & Contours

Data Noise: {{landscape.noise | number:1}} Samples: {{landscape.samples}} Learning Rate: {{(landscape.lr/1000) | number:3}} Iterations: {{landscape.iters}}

Final Loss

After path iterations

Est. w0

Intercept

Est. w1

Slope

True w0

Ground truth

True w1

Ground truth

Interpretation: Gradient descent follows the steepest descent path on the convex surface toward the unique global minimum.

9. LLM-Specific Considerations

Next Token Prediction

Use cross-entropy loss over vocabulary
Compute loss for each token position
Average across sequence length

Label Smoothing

Soften hard targets to prevent overconfidence: $$y'_i = (1 - \epsilon)y_i + \frac{\epsilon}{K}$$ where $\epsilon$ is smoothing parameter, $K$ is number of classes.

Masked Language Modeling (BERT)

Mask random tokens (15% typically)
Predict masked tokens using cross-entropy
Only compute loss on masked positions

Key Takeaways

MSE for regression, sensitive to outliers (quadratic penalty)
MAE for robust regression, linear penalty
Huber loss balances MSE and MAE
Cross-entropy for classification and language modeling
Perplexity is exponentiated cross-entropy
Choose loss function based on task and data characteristics
Gradient descent minimizes loss iteratively