5.1 Parametric Models

Introduction

Parametric models are a fundamental class of machine learning algorithms characterized by a fixed number of parameters. Once trained, these models capture all information from the training data within their parameters, making them computationally efficient and well-suited for large language models (LLMs).

Key Characteristics

Fixed Parameter Count: The number of parameters is determined before training and does not grow with data size
Strong Assumptions: Make explicit assumptions about data distribution (e.g., linearity, normality)
Fast Prediction: Once trained, predictions are computationally inexpensive
Interpretability: Parameters often have clear mathematical interpretations

Mathematical Foundation

A parametric model defines a function $ f_\theta : \mathcal{X} \rightarrow \mathcal{Y} $ where:

$ \theta \in \mathbb{R}^d $ is the parameter vector with fixed dimension $ d $
$ \mathcal{X} $ is the input space
$ \mathcal{Y} $ is the output space

The goal is to find $ \theta^* $ that minimizes the loss function: $$\theta^* = \arg\min_\theta \frac{1}{n}\sum_{i=1}^{n} L(y_i, f_\theta(x_i))$$ where $ L $ measures the difference between predicted and actual values.

Linear Regression: The Simplest Parametric Model

Linear regression assumes the relationship: $$y = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \ldots + \theta_d x_d + \epsilon$$ or in vector form: $ y = \theta^T x + \epsilon $ where $ \epsilon \sim \mathcal{N}(0, \sigma^2) $

Interactive: Linear Regression

Adjust the slope and intercept to see how parameters affect the model:

Slope (θ₁): {{slope}} Intercept (θ₀): {{intercept}}

Mean Squared Error: {{mse | number:3}}

Model Equation: y = {{intercept | number:2}} + {{slope | number:2}}x

Logistic Regression: Classification with Parameters

For binary classification, logistic regression models the probability: $$P(y=1|x) = \sigma(\theta^T x) = \frac{1}{1 + e^{-\theta^T x}}$$ where $ \sigma $ is the sigmoid function.

Interactive: Sigmoid Function

See how the decision boundary changes with different parameters:

Weight (θ₁): {{weight}} Bias (θ₀): {{bias}}

Decision Boundary: x = {{-bias/weight | number:2}} (where P(y=1) = 0.5)

Neural Networks: Parametric Models for LLMs

Modern language models are parametric neural networks with millions to billions of parameters. A simple neural network layer computes: $$h = \sigma(Wx + b)$$ where $ W \in \mathbb{R}^{m \times n} $ and $ b \in \mathbb{R}^m $ are learnable parameters.

Interactive: Single Neuron Activation

Visualize how a single neuron transforms input:

Weight 1: {{w1}} Weight 2: {{w2}} Bias: {{neuronBias}} Activation:

Advantages for LLMs

Scalability: Fixed memory footprint regardless of training data size
Generalization: Parameters capture patterns rather than memorizing examples
Composability: Can stack layers to create deep architectures
Transfer Learning: Pre-trained parameters can be fine-tuned for new tasks

Limitations

Model Bias: Wrong functional form leads to poor performance
Limited Flexibility: Cannot adapt complexity to data
Assumption Dependence: Performance degrades if assumptions are violated

Sample Data for Exploration

x₁	x₂	y (true)	y (predicted)
{{point.x1 \| number:2}}	{{point.x2 \| number:2}}	{{point.y \| number:2}}	{{point.pred \| number:2}}