gen-briefly-explain-the-source-coding-theorem-and-one-common-mis-3000

Briefly explain the source coding theorem and one common misconception about it.

## Source Coding Theorem The **source coding theorem** (Shannon's first theorem) states that data from a source can be compressed down to its **entropy rate** but no further without losing information. Specifically, for a source with entropy H, you can encode messages using an average of H bits per symbol (or arbitrarily close to it), but you cannot reliably compress below this limit. ## Common Misconception **Misconception**: "The source coding theorem means every individual file can be compressed to its entropy." **Reality**: The theorem is an *asymptotic* result that applies to long sequences or many independent samples. Individual files may compress better or worse than the average entropy rate. Some files might be incompressible (already random-looking), while others compress significantly. Only when averaging over many typical sequences from the source do you approach the entropy limit. For example, while English text has an entropy around 1-2 bits per character, a specific short message might require more bits per character to encode, or a repetitive message like "AAAA..." might compress to nearly zero bits per character.