13.4 Gaussian Processes

Introduction to Gaussian Processes

A Gaussian Process (GP) is a powerful tool in machine learning for modeling distributions over functions. Instead of learning the parameters of a specific function (like in linear regression), a GP allows us to reason about the functions themselves. It's a non-parametric approach, meaning its complexity grows with the amount of data.

A GP is defined as a collection of random variables, any finite number of which have a joint Gaussian distribution. This provides a flexible and robust way to perform regression, known as Gaussian Process Regression.

The Model: A Distribution Over Functions

We can think of a function \(f(x)\) as a very long vector, where each entry corresponds to the function's value at a particular point \(x\). A GP defines a prior distribution over these functions.

A GP is fully specified by its mean function \(m(x)\) and its covariance function (or kernel) \(k(x, x')\):

\(f(x) \sim \mathcal{GP}(m(x), k(x, x'))\)

The mean function \(m(x)\) is often assumed to be zero. The kernel \(k(x, x')\) is the crucial part, as it defines the similarity between the function's values at different points \(x\) and \(x'\).

The Kernel Function

The kernel determines the properties of the functions in our distribution (e.g., their smoothness). A common choice is the Radial Basis Function (RBF) kernel, also known as the squared exponential kernel:

\(k(x, x') = \sigma_f^2 \exp\left(-\frac{(x - x')^2}{2l^2}\right)\)

Here, \(\sigma_f^2\) is the signal variance, controlling the overall vertical variation of the function, and \(l\) is the length scale, which determines how quickly the function can vary. A small length scale means the function is "wiggly," while a large length scale results in a smoother function.

Gaussian Process Regression

Given a set of training points \((\mathbf{X}, \mathbf{y})\), we can use the GP prior to compute a posterior distribution over functions that fit these points.

For a set of test points \(\mathbf{X}_*\), the joint distribution of the training outputs \(\mathbf{y}\) and the test outputs \(\mathbf{f}_*\) is given by:

\(\begin{pmatrix} \mathbf{y} \\ \mathbf{f}_* \end{pmatrix} \sim \mathcal{N}\left(\mathbf{0}, \begin{pmatrix} K(\mathbf{X}, \mathbf{X}) + \sigma_n^2\mathbf{I} & K(\mathbf{X}, \mathbf{X}_*) \\ K(\mathbf{X}_*, \mathbf{X}) & K(\mathbf{X}_*, \mathbf{X}_*) \end{pmatrix}\right)\)

where \(K(\mathbf{X}, \mathbf{X})\) is the matrix of kernel evaluations for all pairs of training points, and \(\sigma_n^2\) is the noise variance.

By conditioning the joint distribution on the observed data, we get the predictive distribution for \(\mathbf{f}_*\), which is also a Gaussian:

\(p(\mathbf{f}_* | \mathbf{X}_*, \mathbf{X}, \mathbf{y}) = \mathcal{N}(\boldsymbol{\mu}_*, \boldsymbol{\Sigma}_*)\)

with mean \(\boldsymbol{\mu}_*\) and covariance \(\boldsymbol{\Sigma}_*\):

\(\boldsymbol{\mu}_* = K(\mathbf{X}_*, \mathbf{X}) [K(\mathbf{X}, \mathbf{X}) + \sigma_n^2\mathbf{I}]^{-1} \mathbf{y}\)

\(\boldsymbol{\Sigma}_* = K(\mathbf{X}_*, \mathbf{X}_*) - K(\mathbf{X}_*, \mathbf{X}) [K(\mathbf{X}, \mathbf{X}) + \sigma_n^2\mathbf{I}]^{-1} K(\mathbf{X}, \mathbf{X}_*)\)

The diagonal of \(\boldsymbol{\Sigma}_*\) gives us the variance at each test point, providing a natural measure of uncertainty.