13.2 Bayesian Inference

Introduction

Bayesian inference provides practical methods for computing posterior distributions and making predictions. While conceptually elegant, exact Bayesian inference is often intractable, requiring approximation techniques like MCMC and variational inference.

1. Maximum A Posteriori (MAP) Estimation

1.1 Definition

MAP estimation finds the mode of the posterior:

$$\hat{\theta}_{\text{MAP}} = \arg\max_{\theta} P(\theta | \mathcal{D}) = \arg\max_{\theta} P(\mathcal{D} | \theta) P(\theta)$$

Taking logarithms:

$$\hat{\theta}_{\text{MAP}} = \arg\max_{\theta} \left[\log P(\mathcal{D} | \theta) + \log P(\theta)\right]$$

This is equivalent to regularized maximum likelihood!

1.2 Connection to Regularization

For linear regression with Gaussian likelihood and Gaussian prior:

$$\hat{\mathbf{w}}_{\text{MAP}} = \arg\min_{\mathbf{w}} \left[\|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2 + \lambda\|\mathbf{w}\|^2\right]$$

This is ridge regression! The prior acts as L2 regularization.

With a Laplace prior, we get L1 regularization (Lasso):

$$\hat{\mathbf{w}}_{\text{MAP}} = \arg\min_{\mathbf{w}} \left[\|\mathbf{y} - \mathbf{X}\mathbf{w}\|^2 + \lambda\|\mathbf{w}\|_1\right]$$

1.3 Limitations of MAP

• Provides only a point estimate, ignoring uncertainty
• Can be sensitive to parameterization
• May not generalize well (no model averaging)

2. Laplace Approximation

Approximate the posterior with a Gaussian centered at the MAP estimate.

2.1 Method

1. Find MAP estimate: $\hat{\theta}_{\text{MAP}} = \arg\max_{\theta} \log P(\theta | \mathcal{D})$
2. Compute Hessian at MAP: $\mathbf{H} = -\nabla^2 \log P(\hat{\theta}_{\text{MAP}} | \mathcal{D})$
3. Approximate posterior as Gaussian: $$P(\theta | \mathcal{D}) \approx \mathcal{N}(\hat{\theta}_{\text{MAP}}, \mathbf{H}^{-1})$$

2.2 Advantages and Limitations

Advantages:

• Fast: Only requires finding mode and computing Hessian
• Provides uncertainty estimate (covariance $\mathbf{H}^{-1}$)
• Works well for unimodal, roughly symmetric posteriors

Limitations:

• Poor for multimodal posteriors
• Fails for highly skewed distributions
• Gaussian assumption may be unrealistic

3. Markov Chain Monte Carlo (MCMC)

MCMC methods generate samples from the posterior distribution using Markov chains.

3.1 Metropolis-Hastings Algorithm

Algorithm:

1. Initialize $\theta^{(0)}$
2. For $t = 0, 1, 2, \ldots$:
   • Propose $\theta^* \sim q(\theta^* | \theta^{(t)})$
   • Compute acceptance ratio: $$\alpha = \min\left(1, \frac{P(\theta^* | \mathcal{D}) q(\theta^{(t)} | \theta^*)}{P(\theta^{(t)} | \mathcal{D}) q(\theta^* | \theta^{(t)})}\right)$$
   • Accept $\theta^{(t+1)} = \theta^*$ with probability $\alpha$, else $\theta^{(t+1)} = \theta^{(t)}$

The chain converges to the posterior distribution!

3.2 Gibbs Sampling

For multi-dimensional $\boldsymbol{\theta} = [\theta_1, \ldots, \theta_d]$, sample each component conditionally:

1. Initialize $\boldsymbol{\theta}^{(0)}$
2. For $t = 0, 1, 2, \ldots$:
   • Sample $\theta_1^{(t+1)} \sim P(\theta_1 | \theta_2^{(t)}, \ldots, \theta_d^{(t)}, \mathcal{D})$
   • Sample $\theta_2^{(t+1)} \sim P(\theta_2 | \theta_1^{(t+1)}, \theta_3^{(t)}, \ldots, \theta_d^{(t)}, \mathcal{D})$
   • $\vdots$
   • Sample $\theta_d^{(t+1)} \sim P(\theta_d | \theta_1^{(t+1)}, \ldots, \theta_{d-1}^{(t+1)}, \mathcal{D})$

Effective when conditional distributions are easy to sample.

3.3 Hamiltonian Monte Carlo (HMC)

Uses gradients to propose distant points with high acceptance:

• Introduce momentum variable $\mathbf{r}$
• Define Hamiltonian: $H(\boldsymbol{\theta}, \mathbf{r}) = -\log P(\boldsymbol{\theta}|\mathcal{D}) + \frac{1}{2}\mathbf{r}^T\mathbf{M}^{-1}\mathbf{r}$
• Simulate Hamiltonian dynamics using leapfrog integration
• Accept/reject using Metropolis criterion

HMC is highly efficient and used in modern software like Stan.

4. Variational Inference

Variational inference approximates the posterior by finding the closest distribution from a tractable family.

4.1 Variational Objective

Choose a family of distributions $\mathcal{Q}$ (e.g., factorized Gaussians). Find:

$$q^*(\theta) = \arg\min_{q \in \mathcal{Q}} \text{KL}(q(\theta) \| P(\theta | \mathcal{D}))$$

Since the KL divergence to the posterior is intractable, maximize the Evidence Lower Bound (ELBO):

$$\mathcal{L}(q) = \mathbb{E}_{q(\theta)}[\log P(\mathcal{D} | \theta)] - \text{KL}(q(\theta) \| P(\theta))$$

Maximizing ELBO is equivalent to minimizing KL divergence to the posterior.

4.2 Mean-Field Variational Inference

Assume factorized variational distribution:

$$q(\boldsymbol{\theta}) = \prod_{i=1}^{d} q_i(\theta_i)$$

Coordinate ascent updates:

$$\log q_j(\theta_j) = \mathbb{E}_{q_{-j}}[\log P(\boldsymbol{\theta}, \mathcal{D})] + \text{const}$$

where $q_{-j}$ denotes all factors except $q_j$.

4.3 Automatic Differentiation Variational Inference (ADVI)

Use gradient-based optimization to maximize ELBO:

1. Parameterize $q(\theta; \phi)$ (e.g., Gaussian with mean and variance parameters $\phi$)
2. Estimate gradient: $\nabla_\phi \mathcal{L}(q)$ using Monte Carlo
3. Update parameters: $\phi \leftarrow \phi + \eta \nabla_\phi \mathcal{L}(q)$

ADVI enables black-box inference with automatic differentiation.

5. Model Comparison and Selection

5.1 Marginal Likelihood

The evidence $P(\mathcal{D} | M)$ quantifies how well model $M$ explains the data:

$$P(\mathcal{D} | M) = \int P(\mathcal{D} | \theta, M) P(\theta | M) \, d\theta$$

It naturally trades off fit and complexity (Occam's razor).

5.2 Bayes Factor

Compare models $M_1$ and $M_2$:

$$\text{BF}_{12} = \frac{P(\mathcal{D} | M_1)}{P(\mathcal{D} | M_2)}$$

Interpretation (Jeffrey's scale):

• BF > 100: Decisive evidence for $M_1$
• BF > 10: Strong evidence
• BF > 3: Moderate evidence
• BF ≈ 1: No clear preference

5.3 Computing the Evidence

Exact computation is usually intractable. Approximations:

• Laplace approximation: $P(\mathcal{D}) \approx P(\mathcal{D}|\hat{\theta}_{\text{MAP}}) P(\hat{\theta}_{\text{MAP}}) (2\pi)^{d/2} |\mathbf{H}|^{-1/2}$
• Harmonic mean estimator: From MCMC samples
• Bridge sampling: More accurate MCMC-based method
• Variational lower bound: ELBO provides a lower bound

6. Hierarchical Models

Hierarchical (multi-level) models have parameters at multiple levels.

6.1 Structure

$$\text{Data: } \mathcal{D}_i | \theta_i \sim P(\mathcal{D}_i | \theta_i)$$

$$\text{Group parameters: } \theta_i | \phi \sim P(\theta_i | \phi)$$

$$\text{Hyperparameters: } \phi \sim P(\phi)$$

Information is shared across groups through hyperparameters.

6.2 Advantages

• Partial pooling: Balances individual and group-level effects
• Handles small sample sizes per group
• Models complex dependencies
• Automatic regularization

7. Practical Bayesian Workflow

7.1 Model Building

1. Define likelihood: Choose appropriate distribution for data
2. Specify priors: Encode domain knowledge or use weakly informative priors
3. Check prior predictive: Simulate data from prior to verify reasonableness

7.2 Inference

1. Run MCMC or VI: Obtain posterior samples or approximation
2. Diagnose convergence: Check trace plots, R-hat, effective sample size
3. Assess posterior: Compute summaries, credible intervals

7.3 Model Checking

1. Posterior predictive checks: Simulate data from posterior and compare to observed
2. Cross-validation: LOO-CV for model comparison
3. Sensitivity analysis: Vary priors and check robustness

8. Software Tools

• Stan: Probabilistic programming with HMC
• PyMC: Python library for Bayesian modeling
• JAGS: BUGS-like language for Gibbs sampling
• TensorFlow Probability: Bayesian inference with deep learning
• Pyro: Probabilistic programming on PyTorch
• Edward2: Probabilistic programming in TensorFlow

9. Connection to LLMs

Bayesian inference has profound connections to modern language models:

• Variational autoencoders (VAEs): Use variational inference for text generation
• Bayesian neural networks: Quantify uncertainty in neural predictions
• Dropout as approximate inference: MC dropout approximates Bayesian posterior
• Ensemble methods: Model averaging similar to Bayesian model combination
• Prior through pre-training: Pre-trained weights encode language priors
• Continual learning: Bayesian approaches prevent catastrophic forgetting
• Active learning: Bayesian optimal design for data selection in RLHF
• Calibration: Bayesian methods improve confidence calibration in LLM outputs
• Prompt engineering: Similar to specifying informative priors
• In-context learning: Resembles Bayesian updating from few examples

10. Advanced Topics

10.1 Sequential Monte Carlo (Particle Filters)

For sequential data, update posterior incrementally using particle filtering.

10.2 Approximate Bayesian Computation (ABC)

When likelihood is intractable, simulate data and accept parameters that produce similar data.

10.3 Normalizing Flows

Learn flexible variational distributions using invertible neural networks.

10.4 Amortized Inference

Train inference networks to predict posteriors, enabling fast inference on new data.

Key Takeaways

• MAP estimation provides point estimates with regularization interpretation
• Laplace approximation gives Gaussian posterior approximation around MAP
• MCMC methods (MH, Gibbs, HMC) generate samples from exact posterior
• Variational inference trades exactness for computational efficiency
• Marginal likelihood and Bayes factors enable principled model comparison
• Hierarchical models share information across groups through hyperparameters
• Modern software (Stan, PyMC) makes Bayesian inference accessible
• Bayesian methods provide uncertainty quantification crucial for reliable LLMs
• Variational inference underlies VAEs and other generative models
• Bayesian workflow: model building, inference, and checking form an iterative process