Link copied to clipboard!

3.2 Joint and Conditional Probability

We now study joint distributions of two discrete random variables $(X,Y)$, their marginals, and conditional probabilities. Mastery of these operations is central to modeling multi-token contexts and attention mechanisms in Large Language Models (LLMs), where dependencies between positions matter.

Joint PMF: $p_{X,Y}(x,y) = \Pr(X=x, Y=y)$.

Marginals: $p_X(x) = \sum_y p_{X,Y}(x,y), \quad p_Y(y) = \sum_x p_{X,Y}(x,y)$.

Conditional: $p_{Y|X}(y|x) = \dfrac{p_{X,Y}(x,y)}{p_X(x)}$ (if $p_X(x)>0$).

Bayes Rule: $p_{X|Y}(x|y) = \dfrac{p_{Y|X}(y|x) p_X(x)}{\sum_{x'} p_{Y|X}(y|x') p_X(x')}$.

Independence: $X \perp Y$ iff $p_{X,Y}(x,y) = p_X(x) p_Y(y)$ for all $(x,y)$.

Editable Joint Distribution (Discrete 4×4)

Adjust the raw non-negative weights $w_{ij}$. They are normalized automatically to form a valid joint PMF. Click a heatmap cell or use sliders. Observe marginals and conditional slice.

Weights

{{$parent.$index}},{{$index}}

Sum of weights = {{rawSum | number:3}} → normalized to 1.

Entropy $H(X,Y)=-\sum_{x,y} p_{x,y}\log p_{x,y}$: {{entropy | number:4}}

Mutual Information $I(X;Y)=\sum_{x,y} p_{x,y}\log\dfrac{p_{x,y}}{p_X(x)p_Y(y)}$: {{mutualInfo | number:4}}

(Logs base $e$; change of base factor for bits: divide by $\ln 2$.)

Heatmap of $p_{X,Y}(x,y)$

low

mid

high

Click a cell to select a fixed X or Y index for conditional distribution plot below.

Conditionals & Marginals

Selected mode:

Selected index: {{selectedIndex}}

Mathematical Notes

Normalization: $\sum_{x,y} p_{x,y} = 1$ ensures a valid joint PMF.
Entropy Decomposition: $H(X,Y)=H(X)+H(Y|X)=H(Y)+H(X|Y)$.
Mutual Information (MI): $I(X;Y)=H(X)-H(X|Y)=H(Y)-H(Y|X)\ge 0$ with equality iff independent.
LLM Connection: Attention heads learn structures approximating high $I(X;Y)$ between positions; MI can guide pruning or interpretability.
Conditional Independence: $X \perp Y \mid Z$ generalizes pairwise independence to contexts (critical in masked language modeling).

Bayes Rule Mini Example

Suppose a token class (e.g., punctuation vs word) affects next-token distribution. If $p(X=\text{punct})=0.1$, $p(Y=\text{capital}\mid X=\text{punct})=0.6$, and $p(Y=\text{capital}\mid X=\text{word})=0.05$ then: $$ p(Y=\text{capital}) = 0.6(0.1)+0.05(0.9)=0.105 $$ $$ p(X=\text{punct} \mid Y=\text{capital}) = \frac{0.6(0.1)}{0.105} \approx 0.5714 $$ This mirrors decoding adjustments when preceding classification influences next token likelihood.

LLM Relevance

Joint modeling: Token pairs and multi-head attention implicitly maintain joint structures.
Conditionals: Generation is sequential conditioning: $p(t_k | t_{
Mutual Information: Helps evaluate redundancy vs informative context spans.
Bayesian updates: Fine-tuning integrates new evidence (data) akin to conditional probability re-weighting.

Scan with Phone