gen-define-cross-entropy-and-give-one-short-worked-example-62b3

Define cross-entropy and give one short worked example.

# Cross-Entropy **Cross-entropy** measures the difference between two probability distributions. It quantifies how many bits (or nats) are needed to encode data from a true distribution *p* using a coding scheme optimized for an estimated distribution *q*. ## Definition For discrete distributions: **H(p, q) = -Σ p(x) log q(x)** where the sum is over all possible outcomes *x*. ## Worked Example Suppose we're predicting weather with three outcomes: {sunny, rainy, cloudy}. - **True distribution** p: sunny=1, rainy=0, cloudy=0 (it's actually sunny) - **Predicted distribution** q: sunny=0.7, rainy=0.2, cloudy=0.1 **Cross-entropy:** H(p, q) = -(1 × log(0.7) + 0 × log(0.2) + 0 × log(0.1)) = -log(0.7) = **0.357** nats (or **0.515** bits with log₂) **Interpretation:** The smaller the cross-entropy, the better. If our model predicted sunny=1.0, the cross-entropy would be 0 (perfect prediction). The value 0.515 bits represents the inefficiency of our probabilistic prediction. --- *Note: In machine learning, cross-entropy is commonly used as a loss function for classification tasks.*