13.3 Bayesian Regression

Introduction

In contrast to standard linear regression which finds a single "best" line, Bayesian regression considers a distribution of possible lines. It uses probability to represent uncertainty about the model's parameters. This approach is powerful because it not only gives us a prediction but also a measure of how confident we are in that prediction.

The core idea is to combine our prior beliefs about the parameters with the evidence from the data to form an updated, posterior belief.

The Model

We start with the standard linear model assumption, where the target variable \(y\) is a linear function of the input features \(\mathbf{x}\), with some added Gaussian noise \(\epsilon\).

\(y = \mathbf{w}^T \mathbf{x} + \epsilon\), where \(\epsilon \sim \mathcal{N}(0, \sigma^2)\)

This implies the likelihood of observing \(y\) given \(\mathbf{x}\) and the weights \(\mathbf{w}\) is Gaussian:

\(p(y | \mathbf{x}, \mathbf{w}, \sigma^2) = \mathcal{N}(y | \mathbf{w}^T \mathbf{x}, \sigma^2)\)

Prior Distribution

In a Bayesian setting, we must specify a prior distribution for our parameters. Let's assume a simple Gaussian prior for the weights \(\mathbf{w}\), centered at zero. This reflects a belief that smaller weights are more likely, which helps to prevent overfitting.

\(p(\mathbf{w}) = \mathcal{N}(\mathbf{w} | \mathbf{0}, \alpha^{-1}\mathbf{I})\)

Here, \(\alpha\) is a hyperparameter representing the precision (inverse variance) of the prior. A large \(\alpha\) means we have a strong belief that the weights are close to zero.

Posterior Distribution

Using Bayes' theorem, we combine the likelihood and the prior to get the posterior distribution of the weights given the data \(D = \{(\mathbf{x}_n, y_n)\}\).

\(p(\mathbf{w} | D) \propto p(D | \mathbf{w}) p(\mathbf{w})\)

Since the likelihood and prior are both Gaussian, the posterior is also a Gaussian distribution. For a dataset with inputs \(\mathbf{X}\) and outputs \(\mathbf{y}\), the posterior is:

\(p(\mathbf{w} | \mathbf{X}, \mathbf{y}, \alpha, \sigma^2) = \mathcal{N}(\mathbf{w} | \mathbf{m}_N, \mathbf{S}_N)\)

where the posterior mean \(\mathbf{m}_N\) and covariance \(\mathbf{S}_N\) are:

\(\mathbf{S}_N = (\alpha\mathbf{I} + \frac{1}{\sigma^2}\mathbf{X}^T\mathbf{X})^{-1}\)

\(\mathbf{m}_N = \frac{1}{\sigma^2}\mathbf{S}_N \mathbf{X}^T\mathbf{y}\)

The mean \(\mathbf{m}_N\) represents our best estimate for the weights after seeing the data, and the covariance \(\mathbf{S}_N\) represents our uncertainty in that estimate.

Predictive Distribution

To make a prediction for a new input \(\mathbf{x}_*\), we don't just use the posterior mean of the weights. Instead, we average over all possible weights, weighted by their posterior probability. This is called the predictive distribution.

\(p(y_* | \mathbf{x}_*, D) = \int p(y_* | \mathbf{x}_*, \mathbf{w}) p(\mathbf{w} | D) d\mathbf{w}\)

The result is another Gaussian distribution:

\(p(y_* | \mathbf{x}_*, D) = \mathcal{N}(y_* | \mathbf{m}_N^T \mathbf{x}_*, \sigma_*^2)\)

The variance of this prediction, \(\sigma_*^2\), includes both the uncertainty in the model's weights and the inherent noise in the data:

\(\sigma_*^2 = \sigma^2 + \mathbf{x}_*^T \mathbf{S}_N \mathbf{x}_*\)

This predictive uncertainty is a key advantage of the Bayesian approach. The model tells us where it is uncertain, which is typically in regions far from the training data.