gen-define-mutual-information-and-give-one-short-worked-example-69d8

Define mutual information and give one short worked example.

# Mutual Information **Definition:** Mutual information I(X;Y) measures how much information one random variable reveals about another. It quantifies the reduction in uncertainty about X when Y is observed (and vice versa). $$I(X;Y) = \sum_{x \in X} \sum_{y \in Y} p(x,y) \log_2 \frac{p(x,y)}{p(x)p(y)}$$ Equivalently: $I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)$ where H is entropy. ## Worked Example Consider two binary random variables: - **X**: Whether it's raining (R) or not (N) - **Y**: Whether someone carries an umbrella (U) or not (N) Joint distribution: | | Y=U | Y=N | p(X) | |-------|-----|-----|------| | **X=R** | 0.3 | 0.1 | 0.4 | | **X=N** | 0.1 | 0.5 | 0.6 | | **p(Y)** | 0.4 | 0.6 | | **Calculate I(X;Y):** $$I(X;Y) = \sum \sum p(x,y) \log_2 \frac{p(x,y)}{p(x)p(y)}$$ $$= 0.3\log_2\frac{0.3}{0.4 \times 0.4} + 0.1\log_2\frac{0.1}{0.4 \times 0.6} + 0.1\log_2\frac{0.1}{0.6 \times 0.4} + 0.5\log_2\frac{0.5}{0.6 \times 0.6}$$ $$= 0.3(0.907) + 0.1(-0.585) + 0.1(-0.585) + 0.5(0.474)$$ $$I(X;Y) \approx \textbf{0.39 bits}$$ **Interpretation:** Observing whether someone carries an umbrella reduces uncertainty about rain by 0.39 bits.