9.1 Simple Linear Regression (Regression I)
Introduction
Simple linear regression models relationship between one predictor \(x\) and target \(y\) using a straight line. Despite simplicity, it provides foundation for all regression methods and remains interpretable for understanding relationships between variables.
1. The Linear Model
Model assumes linear relationship: $$y = \beta_0 + \beta_1 x + \epsilon$$ where:
- \(\beta_0\): Intercept (value of \(y\) when \(x = 0\))
- \(\beta_1\): Slope (change in \(y\) per unit change in \(x\))
- \(\epsilon\): Error term (noise, \(\epsilon \sim \mathcal{N}(0, \sigma^2)\))
2. Ordinary Least Squares (OLS)
Find parameters that minimize sum of squared residuals: $$\min_{\beta_0, \beta_1} \sum_{i=1}^{n}(y_i - \beta_0 - \beta_1 x_i)^2$$
Closed-form solutions: $$\beta_1 = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}(x_i - \bar{x})^2}$$ $$\beta_0 = \bar{y} - \beta_1\bar{x}$$
Interactive: OLS Fitting
See how OLS finds best-fit line:
Estimated Slope (β₁)
{{olsMetrics.slope | number:3}}
Estimated Intercept (β₀)
{{olsMetrics.intercept | number:3}}
R²
{{olsMetrics.r2 | number:3}}
3. R-Squared (Coefficient of Determination)
Measures proportion of variance in \(y\) explained by model: $$R^2 = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}} = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2}$$
where:
- \(\text{SS}_{\text{res}} = \sum(y_i - \hat{y}_i)^2\): Residual sum of squares
- \(\text{SS}_{\text{tot}} = \sum(y_i - \bar{y})^2\): Total sum of squares
\(R^2 = 1\): Perfect fit. \(R^2 = 0\): Model no better than mean. \(R^2 < 0\): Model worse than mean.
Interactive: Understanding R²
See how fit quality affects R²:
R² = {{r2Metrics.r2 | number:3}}: {{r2Metrics.interpretation}}
Explained Variance: {{(r2Metrics.r2 * 100) | number:1}}%
4. Residuals
Residual for observation \(i\): $$e_i = y_i - \hat{y}_i$$ Residuals reveal model fit quality and assumptions violations.
Residual Analysis
- Residual Plot: Plot \(e_i\) vs \(\hat{y}_i\). Should show random scatter.
- Patterns indicate problems: Curvature (non-linearity), funnel shape (heteroscedasticity)
- Normality: Histogram or Q-Q plot of residuals should be approximately normal
Interactive: Residual Analysis
Examine residual patterns for different scenarios:
Diagnosis: {{residualDiagnosis}}
5. Assumptions of Linear Regression
1. Linearity
Relationship between \(x\) and \(y\) is linear
2. Independence
Observations are independent (no autocorrelation)
3. Homoscedasticity
Constant variance of errors: \(\text{Var}(\epsilon_i) = \sigma^2\) for all \(i\)
4. Normality
Errors follow normal distribution: \(\epsilon_i \sim \mathcal{N}(0, \sigma^2)\)
5. No Multicollinearity
(Relevant for multiple regression - covered next)
Interactive: Assumption Violations
See impact of violating assumptions:
Impact: {{violationImpact}}
Solution: {{violationSolution}}
6. Prediction and Confidence Intervals
Prediction Interval
For new observation at \(x_0\): $$\hat{y}_0 \pm t_{n-2, \alpha/2} \cdot s_e\sqrt{1 + \frac{1}{n} + \frac{(x_0 - \bar{x})^2}{\sum(x_i - \bar{x})^2}}$$ where \(s_e\) is standard error of regression.
Confidence Interval for Mean
For mean response at \(x_0\): $$\hat{y}_0 \pm t_{n-2, \alpha/2} \cdot s_e\sqrt{\frac{1}{n} + \frac{(x_0 - \bar{x})^2}{\sum(x_i - \bar{x})^2}}$$ Narrower than prediction interval (less uncertainty).
Interactive: Prediction vs Confidence Intervals
Compare interval types:
Confidence Interval: For mean of \(y\) at given \(x\)
Prediction Interval: For individual new observation (wider)
7. Hypothesis Testing
Test for Slope
Null hypothesis: \(H_0: \beta_1 = 0\) (no relationship) Test statistic: $$t = \frac{\hat{\beta}_1}{\text{SE}(\hat{\beta}_1)}$$ follows \(t\)-distribution with \(n-2\) degrees of freedom.
If \(|t| > t_{\text{critical}}\), reject \(H_0\): significant relationship exists.
8. Application to LLMs
- Token Prediction: Simple regression between context length and perplexity
- Training Dynamics: Model loss vs training steps (often log-linear)
- Scaling Laws: Performance vs model size/data size relationships
- Feature Analysis: Understand impact of single features on predictions
9. Optimization & Fitting Visualizations
The following interactive tools illustrate how optimization algorithms find the least-squares solution and how model complexity changes with polynomial features.
Interactive: Gradient Descent Path in Parameter Space
Shows the surface of the Sum of Squared Errors (SSE) across intercept (β₀) and slope (β₁) with the gradient descent trajectory.
Current β₀
{{gd.current.b0 | number:3}}
Current β₁
{{gd.current.b1 | number:3}}
SSE
{{gd.current.sse | number:3}}
Iteration
{{gd.iter}}
Interactive: Least Squares Fitting Animation
Watch the regression line update each iteration of gradient descent while minimizing SSE.
Note: The optimal (closed-form) line is shown in red; the iterative gradient descent estimate is in green.
Interactive: Normal Equation vs Gradient Descent Convergence
Compares closed-form OLS solution with iterative gradient descent convergence in Mean Squared Error.
Dashed line shows final MSE from Normal Equation (instant solution). Gradient descent approaches it over iterations.
Interactive: Polynomial Feature Expansion
Shows how increasing polynomial degree changes the model fit. Higher degree increases flexibility (risk of overfitting).
Train MSE
{{poly.mse | number:3}}
Effective Features
{{poly.degree + 1}}
Interactive: LLM Scaling Law
Visualize power-law relationship between model size and performance:
Scaling Law: Loss ∝ (Model Size)^(-α)
Insight: Larger models consistently improve performance (diminishing returns)
Key Takeaways
- Simple linear regression models \(y = \beta_0 + \beta_1 x + \epsilon\)
- OLS minimizes sum of squared residuals to find best-fit line
- R² measures proportion of variance explained (0 to 1)
- Residual analysis checks model assumptions
- Key assumptions: linearity, independence, homoscedasticity, normality
- Prediction intervals wider than confidence intervals
- Hypothesis testing determines if relationship is significant