9.1 Simple Linear Regression (Regression I)

Introduction

Simple linear regression models relationship between one predictor $x$ and target $y$ using a straight line. Despite simplicity, it provides foundation for all regression methods and remains interpretable for understanding relationships between variables.

1. The Linear Model

Model assumes linear relationship: $$y = \beta_0 + \beta_1 x + \epsilon$$ where:

$\beta_0$: Intercept (value of $y$ when $x = 0$)
$\beta_1$: Slope (change in $y$ per unit change in $x$)
$\epsilon$: Error term (noise, $\epsilon \sim \mathcal{N}(0, \sigma^2)$)

2. Ordinary Least Squares (OLS)

Find parameters that minimize sum of squared residuals: $$\min_{\beta_0, \beta_1} \sum_{i=1}^{n}(y_i - \beta_0 - \beta_1 x_i)^2$$

Closed-form solutions: $$\beta_1 = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}(x_i - \bar{x})^2}$$ $$\beta_0 = \bar{y} - \beta_1\bar{x}$$

Interactive: OLS Fitting

See how OLS finds best-fit line:

Noise Level: {{noiseLevel}}/10 True Slope: {{trueSlope}} Show Residuals

Estimated Slope (β₁)

Estimated Intercept (β₀)

R²

3. R-Squared (Coefficient of Determination)

Measures proportion of variance in $y$ explained by model: $$R^2 = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}} = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2}$$

where:

$\text{SS}_{\text{res}} = \sum(y_i - \hat{y}_i)^2$: Residual sum of squares
$\text{SS}_{\text{tot}} = \sum(y_i - \bar{y})^2$: Total sum of squares

$R^2 = 1$: Perfect fit. $R^2 = 0$: Model no better than mean. $R^2 < 0$: Model worse than mean.

Interactive: Understanding R²

See how fit quality affects R²:

Data Scatter: {{scatter}}/10

R² = {{r2Metrics.r2 | number:3}}: {{r2Metrics.interpretation}}

Explained Variance: {{(r2Metrics.r2 * 100) | number:1}}%

4. Residuals

Residual for observation $i$: $$e_i = y_i - \hat{y}_i$$ Residuals reveal model fit quality and assumptions violations.

Residual Analysis

Residual Plot: Plot $e_i$ vs $\hat{y}_i$. Should show random scatter.
Patterns indicate problems: Curvature (non-linearity), funnel shape (heteroscedasticity)
Normality: Histogram or Q-Q plot of residuals should be approximately normal

Interactive: Residual Analysis

Examine residual patterns for different scenarios:

Data Pattern:

Diagnosis: {{residualDiagnosis}}

5. Assumptions of Linear Regression

1. Linearity

Relationship between $x$ and $y$ is linear

2. Independence

Observations are independent (no autocorrelation)

3. Homoscedasticity

Constant variance of errors: $\text{Var}(\epsilon_i) = \sigma^2$ for all $i$

4. Normality

Errors follow normal distribution: $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$

5. No Multicollinearity

(Relevant for multiple regression - covered next)

Interactive: Assumption Violations

See impact of violating assumptions:

Violation Type:

Impact: {{violationImpact}}

Solution: {{violationSolution}}

6. Prediction and Confidence Intervals

Prediction Interval

For new observation at $x_0$: $$\hat{y}_0 \pm t_{n-2, \alpha/2} \cdot s_e\sqrt{1 + \frac{1}{n} + \frac{(x_0 - \bar{x})^2}{\sum(x_i - \bar{x})^2}}$$ where $s_e$ is standard error of regression.

Confidence Interval for Mean

For mean response at $x_0$: $$\hat{y}_0 \pm t_{n-2, \alpha/2} \cdot s_e\sqrt{\frac{1}{n} + \frac{(x_0 - \bar{x})^2}{\sum(x_i - \bar{x})^2}}$$ Narrower than prediction interval (less uncertainty).

Interactive: Prediction vs Confidence Intervals

Compare interval types:

Confidence Level: {{confidence}}% Confidence Interval Prediction Interval

Confidence Interval: For mean of $y$ at given $x$

Prediction Interval: For individual new observation (wider)

7. Hypothesis Testing

Test for Slope

Null hypothesis: $H_0: \beta_1 = 0$ (no relationship) Test statistic: $$t = \frac{\hat{\beta}_1}{\text{SE}(\hat{\beta}_1)}$$ follows $t$-distribution with $n-2$ degrees of freedom.

If $|t| > t_{\text{critical}}$, reject $H_0$: significant relationship exists.

8. Application to LLMs

Token Prediction: Simple regression between context length and perplexity
Training Dynamics: Model loss vs training steps (often log-linear)
Scaling Laws: Performance vs model size/data size relationships
Feature Analysis: Understand impact of single features on predictions

9. Optimization & Fitting Visualizations

The following interactive tools illustrate how optimization algorithms find the least-squares solution and how model complexity changes with polynomial features.

Interactive: Gradient Descent Path in Parameter Space

Shows the surface of the Sum of Squared Errors (SSE) across intercept (β₀) and slope (β₁) with the gradient descent trajectory.

Learning Rate (η): {{gd.lr | number:3}} Iterations: {{gd.maxIter}} Init β₀: {{gd.initB0}} Init β₁: {{gd.initB1}}

Current β₀

Current β₁

SSE

Iteration

Interactive: Least Squares Fitting Animation

Watch the regression line update each iteration of gradient descent while minimizing SSE.

Learning Rate: {{lineAnim.lr | number:3}} Noise: {{lineAnim.noise}} Points: {{lineAnim.n}}

Note: The optimal (closed-form) line is shown in red; the iterative gradient descent estimate is in green.

Interactive: Normal Equation vs Gradient Descent Convergence

Compares closed-form OLS solution with iterative gradient descent convergence in Mean Squared Error.

Learning Rate: {{conv.lr | number:3}} Iterations: {{conv.maxIter}}

Dashed line shows final MSE from Normal Equation (instant solution). Gradient descent approaches it over iterations.

Interactive: Polynomial Feature Expansion

Shows how increasing polynomial degree changes the model fit. Higher degree increases flexibility (risk of overfitting).

Degree: {{poly.degree}} Noise: {{poly.noise}} Points: {{poly.n}}

Train MSE

Effective Features

Interactive: LLM Scaling Law

Visualize power-law relationship between model size and performance:

Model Sizes (Millions): Up to {{maxSize}}M

Scaling Law: Loss ∝ (Model Size)^(-α)

Insight: Larger models consistently improve performance (diminishing returns)

Key Takeaways

Simple linear regression models $y = \beta_0 + \beta_1 x + \epsilon$
OLS minimizes sum of squared residuals to find best-fit line
R² measures proportion of variance explained (0 to 1)
Residual analysis checks model assumptions
Key assumptions: linearity, independence, homoscedasticity, normality
Prediction intervals wider than confidence intervals
Hypothesis testing determines if relationship is significant