gen-define-conditional-entropy-and-give-one-short-worked-example-6d8a

Define conditional entropy and give one short worked example.

# Conditional Entropy ## Definition **Conditional entropy** H(Y|X) measures the average uncertainty remaining about random variable Y after observing random variable X. **Formula:** $$H(Y|X) = \sum_{x \in X} P(x) \cdot H(Y|X=x) = -\sum_{x \in X} \sum_{y \in Y} P(x,y) \log_2 P(y|x)$$ Interpretation: It's the expected entropy of Y, weighted by the probability distribution of X. ## Worked Example Consider a simple communication system: - X = transmitted bit (0 or 1), each with probability 0.5 - Y = received bit, with 10% error rate **Joint probability table:** | | Y=0 | Y=1 | |-------|------|------| | **X=0** | 0.45 | 0.05 | | **X=1** | 0.05 | 0.45 | **Calculation:** For X=0: P(Y=0|X=0) = 0.9, P(Y=1|X=0) = 0.1 $$H(Y|X=0) = -0.9\log_2(0.9) - 0.1\log_2(0.1) = 0.469 \text{ bits}$$ For X=1: P(Y=0|X=1) = 0.1, P(Y=1|X=1) = 0.9 $$H(Y|X=1) = -0.1\log_2(0.1) - 0.9\log_2(0.9) = 0.469 \text{ bits}$$ $$H(Y|X) = 0.5(0.469) + 0.5(0.469) = \boxed{0.469 \text{ bits}}$$ This means after observing the transmitted bit, about 0.47 bits of uncertainty remain about the received bit (due to noise).