gen-define-shannon-entropy-and-give-one-short-worked-example-d387

Define Shannon entropy and give one short worked example.

# Shannon Entropy **Definition:** Shannon entropy measures the average uncertainty or information content in a random variable. For a discrete random variable X with possible values {x₁, x₂, ..., xₙ} and probability mass function p(x), the entropy is: $$H(X) = -\sum_{i=1}^{n} p(x_i) \log_2 p(x_i)$$ The unit is **bits** (when using log base 2). Higher entropy means more uncertainty/randomness. ## Worked Example **Question:** What is the entropy of a fair coin flip? **Solution:** - X = {Heads, Tails} - p(Heads) = 0.5, p(Tails) = 0.5 $$H(X) = -[p(H)\log_2 p(H) + p(T)\log_2 p(T)]$$ $$H(X) = -[0.5 \log_2(0.5) + 0.5 \log_2(0.5)]$$ $$H(X) = -[0.5(-1) + 0.5(-1)]$$ $$H(X) = -[-0.5 - 0.5] = 1 \text{ bit}$$ **Interpretation:** A fair coin has maximum uncertainty (1 bit of entropy), meaning you need exactly 1 bit of information to specify the outcome.