gen-define-the-chain-rule-of-entropy-and-give-one-short-worked-e-9ddc

Define the chain rule of entropy and give one short worked example.

# Chain Rule of Entropy ## Definition The **chain rule of entropy** states that the joint entropy of multiple random variables can be decomposed as a sum of conditional entropies: $$H(X_1, X_2, \ldots, X_n) = \sum_{i=1}^{n} H(X_i \mid X_1, \ldots, X_{i-1})$$ For two variables, this simplifies to: $$H(X, Y) = H(X) + H(Y \mid X)$$ This can also be written as: $$H(X, Y) = H(Y) + H(X \mid Y)$$ ## Worked Example Consider a fair coin flip $X$ and a second variable $Y$ where: - $X \in \{H, T\}$ with $P(X=H) = P(X=T) = 1/2$ - $Y = X$ (i.e., $Y$ always equals $X$) **Calculate $H(X,Y)$ using the chain rule:** $$H(X, Y) = H(X) + H(Y \mid X)$$ - $H(X) = -\sum P(x) \log_2 P(x) = -2 \cdot \frac{1}{2}\log_2\frac{1}{2} = 1$ bit - $H(Y \mid X) = 0$ because once we know $X$, we know $Y$ with certainty Therefore: $$H(X, Y) = 1 + 0 = 1 \text{ bit}$$ This makes sense: the pair $(X,Y)$ contains the same information as $X$ alone since $Y$ is redundant.