gen-explain-the-source-coding-theorem-as-you-would-to-a-capable-279b

Explain the source coding theorem as you would to a capable student, in 3-4 sentences.

The source coding theorem tells us the fundamental limit for how much we can compress data without losing information. Specifically, if a source produces messages with entropy H (measuring their average uncertainty in bits), you need at least H bits per symbol to encode those messages reliably, but you can get arbitrarily close to H with clever encoding. For example, if some letters appear much more frequently than others, you can assign shorter codes to common letters (like Morse code does), achieving compression that approaches the entropy limit as your message gets longer. Any attempt to compress below H bits per symbol will necessarily lose information.