gen-define-self-information-and-give-one-short-worked-example-f11d

Define self-information and give one short worked example.

# Self-Information **Definition:** Self-information (or surprisal) measures the amount of information gained when observing a specific outcome of a random variable. For an event $A$ with probability $P(A)$, the self-information is: $$I(A) = -\log_2 P(A) = \log_2 \frac{1}{P(A)}$$ measured in bits (when using log base 2). **Key intuition:** Rare events carry more information than common events. ## Worked Example Consider flipping a fair coin that lands heads. - Probability: $P(\text{Heads}) = 0.5$ - Self-information: $$I(\text{Heads}) = -\log_2(0.5) = \log_2(2) = 1 \text{ bit}$$ This means observing heads from a fair coin provides exactly **1 bit** of information. **Contrast:** If the coin were biased with $P(\text{Heads}) = 0.99$: $$I(\text{Heads}) = -\log_2(0.99) \approx 0.014 \text{ bits}$$ The common outcome (heads from a biased coin) provides very little information, while the rare outcome would provide much more.