gen-define-joint-entropy-and-give-one-short-worked-example-8bce

Define joint entropy and give one short worked example.

# Joint Entropy ## Definition **Joint entropy** H(X,Y) measures the uncertainty (or information content) in two random variables X and Y considered together. It quantifies the average information needed to specify both variables. **Formula:** $$H(X,Y) = -\sum_{x \in X} \sum_{y \in Y} p(x,y) \log_2 p(x,y)$$ where p(x,y) is the joint probability of X=x and Y=y occurring together. ## Worked Example Consider two binary random variables representing coin flips: - X: Result of first coin (H or T) - Y: Result of second coin (H or T) **Joint probability distribution:** | X\Y | H | T | |-----|-----|-----| | **H** | 1/4 | 1/4 | | **T** | 1/4 | 1/4 | **Calculation:** $$H(X,Y) = -\sum_{x,y} p(x,y) \log_2 p(x,y)$$ $$= -4 \times \frac{1}{4} \log_2\left(\frac{1}{4}\right)$$ $$= -1 \times (-2) = 2 \text{ bits}$$ **Interpretation:** We need 2 bits of information on average to specify the outcome of both coin flips, which makes intuitive sense: 1 bit per independent coin.