9.2 Multiple Linear Regression and Interpretation

Introduction

Multiple linear regression extends simple regression to multiple predictors. Model captures how multiple features jointly influence target. Essential for understanding complex relationships where outcome depends on multiple factors.

1. The Multiple Regression Model

Model with $p$ predictors: $$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p + \epsilon$$

Matrix form: $$\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}$$ where:

$\mathbf{y}$: $n \times 1$ response vector
$\mathbf{X}$: $n \times (p+1)$ design matrix (includes column of 1s for intercept)
$\boldsymbol{\beta}$: $(p+1) \times 1$ coefficient vector
$\boldsymbol{\epsilon}$: $n \times 1$ error vector

2. OLS Solution

Minimize sum of squared residuals: $$\min_{\boldsymbol{\beta}} ||\mathbf{y} - \mathbf{X}\boldsymbol{\beta}||^2$$

Closed-form solution (normal equations): $$\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$

Requires $\mathbf{X}^T\mathbf{X}$ to be invertible (no perfect multicollinearity).

Interactive: Multiple Predictors

Visualize regression with two predictors:

Feature 1 Weight: {{weight1}} Feature 2 Weight: {{weight2}} Noise Level: {{multiNoise}}/10

β₁ (Feature 1)

β₂ (Feature 2)

R²

3. Coefficient Interpretation

$\beta_j$ represents change in $y$ for one-unit increase in $x_j$, holding all other predictors constant.

Partial effect: Unlike simple regression, coefficient captures effect after accounting for other variables. This "holding others constant" is key distinction.

Example: House Prices

$$\text{Price} = \beta_0 + \beta_1 \cdot \text{Size} + \beta_2 \cdot \text{Bedrooms}$$

$\beta_1 = 100$: Each additional sq ft increases price by $100, for houses with same number of bedrooms
$\beta_2 = 5000$: Each additional bedroom adds $5000, for houses of same size

Interactive: Partial Effects

See how coefficients represent partial effects:

Fix Feature 2 at: {{fixedValue}}

Interpretation: Slope shows effect of Feature 1 when Feature 2 held constant at {{fixedValue}}

4. Adjusted R²

Regular R² always increases when adding predictors (even if they're useless). Adjusted R² penalizes model complexity: $$R^2_{\text{adj}} = 1 - \frac{(1 - R^2)(n - 1)}{n - p - 1}$$ where $n$ = sample size, $p$ = number of predictors.

Use adjusted R² for model comparison with different numbers of predictors.

Interactive: R² vs Adjusted R²

See how adding predictors affects metrics:

Number of Predictors: {{numPredictors}} Useful Predictors: {{usefulPredictors}}

Observation: R² increases with more predictors, but adjusted R² may decrease if predictors are weak

5. Multicollinearity

Multicollinearity occurs when predictors are highly correlated with each other. Makes coefficient estimates unstable and difficult to interpret.

Problems

Large standard errors: Coefficients have high variance
Unstable estimates: Small data changes cause large coefficient changes
Difficult interpretation: Hard to isolate individual effects
Sign flips: Coefficients may have unexpected signs

Detection: Variance Inflation Factor (VIF)

For predictor $x_j$, regress it on all other predictors: $$\text{VIF}_j = \frac{1}{1 - R^2_j}$$ where $R^2_j$ is R² from regression of $x_j$ on other predictors.

$\text{VIF} < 5$: No concern
$5 \leq \text{VIF} < 10$: Moderate multicollinearity
$\text{VIF} \geq 10$: High multicollinearity - problematic

Interactive: Multicollinearity Impact

See how correlation between predictors affects estimates:

Correlation between Features: {{correlation | number:2}}

VIF (Feature 1)

VIF (Feature 2)

Std Error (β₁)

6. Solutions to Multicollinearity

Remove correlated predictors: Drop one of highly correlated pair
Combine predictors: Create composite variable (e.g., PCA)
Regularization: Ridge regression handles multicollinearity well (next section)
Collect more data: Larger sample can help stabilize estimates

7. Feature Selection

Choosing which predictors to include in model.

Methods

Forward selection: Start with no predictors, add one at a time
Backward elimination: Start with all predictors, remove one at a time
Stepwise: Combination of forward and backward
Best subset: Try all possible combinations (computationally expensive)

Criteria

AIC (Akaike Information Criterion): $\text{AIC} = 2p - 2\ln(L)$
BIC (Bayesian Information Criterion): $\text{BIC} = p\ln(n) - 2\ln(L)$
Lower is better. BIC penalizes complexity more than AIC.

Interactive: Forward Selection

Watch forward selection algorithm in action:

Current Model: {{forwardStatus.features}}

R²: {{forwardStatus.r2 | number:3}}

Adjusted R²: {{forwardStatus.adjR2 | number:3}}

8. Categorical Predictors

Encode categorical variables using dummy variables (one-hot encoding).

Example: Color (Red, Blue, Green)

Create $k-1$ dummy variables for $k$ categories:

$D_{\text{Blue}} = 1$ if Blue, 0 otherwise
$D_{\text{Green}} = 1$ if Green, 0 otherwise
Red is reference category (both dummies = 0)

Model: $y = \beta_0 + \beta_1 D_{\text{Blue}} + \beta_2 D_{\text{Green}}$

Interpretation: $\beta_1$ is difference between Blue and Red, holding other variables constant.

Interactive: Dummy Variables

See how categorical variables are encoded:

Category Effect: {{categoryEffect}}

Coefficients:

β₀ (Red - reference) = {{dummyMetrics.beta0 | number:2}}

β₁ (Blue vs Red) = {{dummyMetrics.beta1 | number:2}}

β₂ (Green vs Red) = {{dummyMetrics.beta2 | number:2}}

9. Interaction Terms

Interaction captures when effect of one predictor depends on value of another: $$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 (x_1 \cdot x_2)$$

Effect of $x_1$ on $y$: $\beta_1 + \beta_3 x_2$ (depends on $x_2$)

Example

Temperature and humidity predicting comfort. Interaction term captures that high temperature feels worse with high humidity.

Interactive: Interaction Effects

Explore interaction between two predictors:

Interaction Strength: {{interactionStr}}

Interpretation: Lines with different slopes indicate interaction

No interaction: Parallel lines

11. Residual Diagnostics (Advanced)

Residual analysis helps validate linear model assumptions (linearity, homoscedasticity, independence, normality). Patterns in residuals indicate model misspecification.

Scenario: Points: {{diag.n}} Show Residual Density Bar

Interpretation Tips:

Random scatter around zero → model form appropriate.
Curved pattern → add polynomial / interaction terms.
Fan shape (increasing spread) → heteroscedasticity; consider transformation or weighted least squares.
Histogram far from bell-shaped → residuals non-normal; affects inference (t-tests), less critical for prediction with large n.

12. Gradient Descent Coefficient Convergence

Compare iterative gradient descent updates of coefficients with OLS optimum for a 2-predictor model. Observe how learning rate impacts convergence speed and stability.

Learning Rate η: {{coefGD.lr | number:3}} Iterations: {{coefGD.maxIter}} Noise: {{coefGD.noise}}

β₀

β₁

β₂

MSE

Iter

Blue path shows (β₁, β₂) updates. MSE curve approaches the OLS minimum. Too large η may cause divergence oscillations.

10. Application to LLMs

Multi-feature embeddings: Token embeddings as multiple predictors for next-token probability
Attention mechanisms: Weighted combination of multiple context vectors
Layer outputs: Each layer output depends on multiple previous layer features
Multimodal models: Combining text, image, audio features
Feature interactions: Cross-attention captures interactions between features

Key Takeaways

Multiple regression models relationship between multiple predictors and outcome
Coefficients represent partial effects (holding others constant)
Adjusted R² better for comparing models with different predictor counts
Multicollinearity makes estimates unstable - detect with VIF
Feature selection balances fit and complexity
Categorical predictors require dummy variable encoding
Interaction terms capture when effects depend on other variables