13.1 Introduction to Bayesian Statistics

Introduction

Bayesian statistics provides a principled framework for reasoning under uncertainty by treating parameters as random variables. Unlike frequentist methods that treat parameters as fixed unknowns, Bayesian inference updates beliefs about parameters as new data arrives.

1. Fundamental Principles

1.1 Probability as Degree of Belief

In the Bayesian framework, probability represents degree of belief or subjective confidence, not just long-run frequency. This allows us to:

• Assign probabilities to hypotheses and parameters
• Update beliefs systematically based on evidence
• Make decisions under uncertainty
• Incorporate prior knowledge naturally

1.2 Bayes' Theorem

The foundation of Bayesian inference is Bayes' theorem:

$$P(H|D) = \frac{P(D|H) \cdot P(H)}{P(D)}$$

In the context of parameter estimation with parameter $\theta$ and data $\mathcal{D}$:

$$P(\theta | \mathcal{D}) = \frac{P(\mathcal{D} | \theta) \cdot P(\theta)}{P(\mathcal{D})}$$

where:

• $P(\theta | \mathcal{D})$ is the posterior distribution (updated belief after seeing data)
• $P(\mathcal{D} | \theta)$ is the likelihood (probability of data given parameter)
• $P(\theta)$ is the prior distribution (initial belief before seeing data)
• $P(\mathcal{D})$ is the evidence or marginal likelihood

1.3 The Evidence

The evidence is computed by marginalizing over all possible parameter values:

$$P(\mathcal{D}) = \int P(\mathcal{D} | \theta) P(\theta) \, d\theta$$

While often intractable, it normalizes the posterior and plays a crucial role in model comparison.

2. Prior Distributions

2.1 Role of the Prior

The prior $P(\theta)$ encodes our knowledge or assumptions about $\theta$ before observing data. It can be:

• Informative: Strong beliefs based on domain knowledge or previous studies
• Weakly informative: Gentle constraints that guide inference
• Non-informative (vague): Minimal influence, letting data dominate

2.2 Common Prior Distributions

For continuous parameters on ℝ:

• Gaussian: $P(\theta) = \mathcal{N}(\mu_0, \sigma_0^2)$
• Uniform: $P(\theta) = \text{Uniform}(a, b)$
• Laplace: $P(\theta) = \text{Laplace}(\mu, b)$

For positive parameters (e.g., variance):

• Inverse-Gamma: $P(\sigma^2) = \text{IG}(\alpha, \beta)$
• Gamma: $P(\lambda) = \text{Gamma}(\alpha, \beta)$
• Half-Cauchy: $P(\sigma) = \text{Half-Cauchy}(\gamma)$

For probabilities in [0,1]:

• Beta: $P(p) = \text{Beta}(\alpha, \beta)$
• Uniform: $P(p) = \text{Uniform}(0, 1)$ (special case Beta(1,1))

2.3 Jeffrey's Prior

A non-informative prior based on the Fisher information:

$$P(\theta) \propto \sqrt{\det \mathcal{I}(\theta)}$$

where $\mathcal{I}(\theta)$ is the Fisher information matrix. It is invariant under reparameterization.

3. Posterior Inference

3.1 Posterior Distribution

After observing data $\mathcal{D} = \{x_1, \ldots, x_n\}$, we update our belief:

$$P(\theta | \mathcal{D}) \propto P(\mathcal{D} | \theta) P(\theta)$$

The posterior combines prior knowledge with data evidence.

3.2 Point Estimates

Common point estimates from the posterior:

• Posterior mean: $\hat{\theta} = \mathbb{E}[\theta | \mathcal{D}] = \int \theta P(\theta|\mathcal{D}) d\theta$
• Posterior mode (MAP): $\hat{\theta}_{\text{MAP}} = \arg\max_\theta P(\theta | \mathcal{D})$
• Posterior median: Value such that $P(\theta \leq \hat{\theta}_{\text{med}} | \mathcal{D}) = 0.5$

3.3 Credible Intervals

A 95% credible interval $[a, b]$ satisfies:

$$P(a \leq \theta \leq b | \mathcal{D}) = 0.95$$

Interpretation: "There is a 95% probability that $\theta$ lies in $[a, b]$ given the data."

Contrast with frequentist confidence intervals, which have a different interpretation.

4. Conjugate Priors

A prior is conjugate to a likelihood if the posterior belongs to the same family as the prior.

4.1 Gaussian Likelihood with Gaussian Prior

Likelihood: $x_i | \mu \sim \mathcal{N}(\mu, \sigma^2)$ (known variance)

Prior: $\mu \sim \mathcal{N}(\mu_0, \sigma_0^2)$

Posterior: $\mu | \mathcal{D} \sim \mathcal{N}(\mu_n, \sigma_n^2)$ where:

$$\mu_n = \frac{\sigma^2 \mu_0 + n\sigma_0^2 \bar{x}}{\sigma^2 + n\sigma_0^2}, \quad \sigma_n^2 = \frac{\sigma^2 \sigma_0^2}{\sigma^2 + n\sigma_0^2}$$

4.2 Bernoulli Likelihood with Beta Prior

Likelihood: $x_i | p \sim \text{Bernoulli}(p)$

Prior: $p \sim \text{Beta}(\alpha, \beta)$

Posterior: $p | \mathcal{D} \sim \text{Beta}(\alpha + \sum x_i, \beta + n - \sum x_i)$

4.3 Poisson Likelihood with Gamma Prior

Likelihood: $x_i | \lambda \sim \text{Poisson}(\lambda)$

Prior: $\lambda \sim \text{Gamma}(\alpha, \beta)$

Posterior: $\lambda | \mathcal{D} \sim \text{Gamma}(\alpha + \sum x_i, \beta + n)$

5. Predictive Distributions

5.1 Posterior Predictive Distribution

To predict a new observation $x_{n+1}$, marginalize over the posterior:

$$P(x_{n+1} | \mathcal{D}) = \int P(x_{n+1} | \theta) P(\theta | \mathcal{D}) \, d\theta$$

This accounts for parameter uncertainty!

5.2 Prior Predictive Distribution

Before seeing data, the predictive distribution is:

$$P(x) = \int P(x | \theta) P(\theta) \, d\theta$$

Useful for prior elicitation and model checking.

6. Bayesian Model Comparison

6.1 Bayes Factor

To compare models $M_1$ and $M_2$:

$$\text{BF}_{12} = \frac{P(\mathcal{D} | M_1)}{P(\mathcal{D} | M_2)} = \frac{\int P(\mathcal{D}|\theta_1, M_1) P(\theta_1|M_1) d\theta_1}{\int P(\mathcal{D}|\theta_2, M_2) P(\theta_2|M_2) d\theta_2}$$

Interpretation:

• BF > 10: Strong evidence for $M_1$
• BF > 3: Moderate evidence for $M_1$
• BF ≈ 1: No clear preference

6.2 Posterior Model Probabilities

Given prior probabilities $P(M_i)$:

$$P(M_i | \mathcal{D}) = \frac{P(\mathcal{D} | M_i) P(M_i)}{\sum_j P(\mathcal{D} | M_j) P(M_j)}$$

7. Bayesian vs Frequentist

7.1 Key Differences

Aspect	Frequentist	Bayesian
Parameters	Fixed, unknown constants	Random variables with distributions
Probability	Long-run frequency	Degree of belief
Prior knowledge	Not explicitly incorporated	Explicitly encoded as prior
Intervals	Confidence intervals	Credible intervals
Model comparison	Hypothesis testing, p-values	Bayes factors, posterior probabilities

8. Connection to LLMs

Bayesian principles are fundamental to modern language models:

• Bayesian neural networks: Treat weights as random variables with distributions
• Uncertainty quantification: Bayesian approaches provide confidence in predictions
• Prior through pre-training: Pre-trained models encode prior knowledge about language
• Fine-tuning as posterior update: Adapting to new data is Bayesian updating
• Dropout as Bayesian approximation: Monte Carlo dropout approximates posterior
• Variational inference: Used in VAEs and probabilistic models for text
• Active learning: Bayesian optimal experimental design for data selection
• Calibration: Bayesian methods improve probability calibration in LLM outputs

9. Practical Considerations

Advantages

• Principled handling of uncertainty
• Natural incorporation of prior knowledge
• Coherent framework for inference and decision-making
• Automatic regularization through priors
• Works well with small data

Challenges

• Choice of prior can be subjective
• Computational complexity (intractable integrals)
• Sensitivity to model misspecification
• Requires expertise in probability and inference

Key Takeaways

• Bayesian statistics treats parameters as random variables with distributions
• Bayes' theorem provides the update rule: posterior ∝ likelihood × prior
• Priors encode beliefs; posteriors update beliefs with data
• Conjugate priors yield closed-form posteriors for certain likelihood families
• Credible intervals have direct probabilistic interpretation
• Predictive distributions marginalize over parameter uncertainty
• Bayes factors provide principled model comparison
• Bayesian framework underlies many modern ML techniques including uncertainty in LLMs