gen-define-the-kullback-leibler-divergence-and-give-one-short-wo-8376

Define the Kullback-Leibler divergence and give one short worked example.

# Kullback-Leibler Divergence ## Definition The **Kullback-Leibler (KL) divergence** measures how one probability distribution differs from a reference probability distribution. For discrete distributions P and Q: $$D_{KL}(P \| Q) = \sum_{x} P(x) \log\frac{P(x)}{Q(x)}$$ For continuous distributions: $$D_{KL}(P \| Q) = \int_{-\infty}^{\infty} p(x) \log\frac{p(x)}{q(x)} dx$$ **Key properties:** - Always non-negative: $D_{KL}(P \| Q) \geq 0$ - Equals zero iff P = Q (almost everywhere) - **Not symmetric**: $D_{KL}(P \| Q) \neq D_{KL}(Q \| P)$ in general ## Worked Example Consider two discrete distributions over outcomes {A, B, C}: - **P**: (0.5, 0.3, 0.2) - **Q**: (0.4, 0.4, 0.2) Calculate $D_{KL}(P \| Q)$: $$D_{KL}(P \| Q) = 0.5 \log\frac{0.5}{0.4} + 0.3 \log\frac{0.3}{0.4} + 0.2 \log\frac{0.2}{0.2}$$ $$= 0.5 \log(1.25) + 0.3 \log(0.75) + 0.2 \log(1)$$ Using natural logarithm: $$= 0.5(0.223) + 0.3(-0.288) + 0.2(0)$$ $$= 0.112 - 0.086 + 0 = \boxed{0.026 \text{ nats}}$$ This small positive value indicates P and Q are fairly similar distributions.