A Mathematical Theory of Communication
A reader's summary of Claude Shannon's 1948 paper — its history, author, main ideas, critiques, and afterlife. The prequel to Attention Is All You Need. Editorial synthesis, not a substitute for the paper.
The paper at a glance
Published in two parts in the Bell System Technical Journal, the paper did something almost unreasonable: it defined information, measured it, and proved the fundamental limits of storing and transmitting it — compression bounded by entropy, communication bounded by channel capacity — in one stroke, with the mathematics essentially finished on arrival. Where most fields grow from decades of partial results, information theory was born adult. It is on the short list for the most consequential single paper of the twentieth century: everything digital — including the model summarizing it — operates inside the limits it drew.
The author
Claude Shannon (1916–2001) had already written what is often called the best master's thesis in history (1937), showing that Boolean algebra could be implemented in relay circuits — the founding insight of digital logic. At Bell Labs during the war he worked on cryptography and fire control, crossed paths with Turing over tea, and quietly assembled the communication theory on his own time. He was also the century's great scientific tinkerer: a juggling, unicycling builder of chess machines, maze-solving mice and a mechanical flame-throwing trumpet — a reminder that play and depth are not opposites. He spent his later decades at MIT, largely declining the fame his own field offered him.
History and context
Bell Labs needed to know how much a telephone line could carry; Nyquist and Hartley had sketched pieces in the 1920s. Shannon's wartime cryptography work — published separately in 1949 and proving the one-time pad unbreakable — was the same theory viewed in a mirror: a cipher is a channel designed to have zero capacity for the eavesdropper. In 1949 Warren Weaver's popularization turned “A” mathematical theory into “The” — and by the mid-1950s the concepts had spread so promiscuously (to psychology, art, biology, management) that Shannon himself wrote the famous editorial “The Bandwagon,” begging the world to stop applying his theorems to things that were not channels.
Main ideas
The bit
Information measured in binary digits — the paper names the bit and makes information a quantity, like energy or mass, independent of what the message means.
Entropy
H = −Σ p log p: the information content of a source is its unpredictability. A fair coin carries one bit per toss; a loaded one carries less; English text, heavily redundant, carries roughly one bit per letter.
The source coding theorem
Entropy is the hard floor of compression: a source can be encoded down to H bits per symbol on average, and no further. Every ZIP file, JPEG and MP3 lives in the room this theorem built.
Channel capacity
Every noisy channel has a capacity C; below it, error-free communication is possible with clever coding — a result so counterintuitive engineers initially refused it. Noise does not doom communication; it only taxes it.
Separation of meaning
'Semantic aspects of communication are irrelevant to the engineering problem.' The scandalous simplification that made the theory possible — and the standing question mark over every application of it to minds.
Redundancy as armor
Languages carry redundancy so messages survive noise — the reason crosswords work, and the principle behind every error-correcting code from CDs to QR codes to spacecraft telemetry.
Critique
- Meaning is amputated, not solved. The engineering triumph came from ignoring semantics; every attempt since to build a “semantic information theory” on top has remained partial. The theory tells you how many bits a message needs — never what it is worth.
- Existence, not construction. The channel coding theorem proves good codes exist without exhibiting them; it took engineering half a century — Hamming, Reed–Solomon, turbo codes, LDPC — to approach the limits Shannon drew in 1948.
- The bandwagon problem. As Shannon warned, the vocabulary travels better than the theorems: much use of “information” and “entropy” in social science and pop-science is metaphor wearing mathematics.
- Average-case only. Entropy speaks of ensembles; the information in one specific object needed a different theory — Kolmogorov complexity, built decades later in explicit homage.
Impact
The theory is the invisible infrastructure of modernity: every modem handshake, JPEG, MP3, QR code, hard-drive read, 5G frame and deep-space photograph is an application. Cryptography became a science on its foundations; genetics adopted its vocabulary the year DNA's structure was found; and machine learning is its newest province — training a language model is, at bottom, minimizing cross-entropy, which is to say: compression. The line from this paper to the Transformer is direct — a large language model is a Shannon source model of human text, scaled until it started talking back; LeCun's dissent is, in these terms, an argument about which source to model.
Notable engagements
- Warren Weaver — the 1949 book-length popularization that carried the theory beyond engineering.
- Kolmogorov — the century's other great probabilist built algorithmic information theory and ranked Shannon's intuition with the greatest of mathematicians.
- Hamming and the coders — fifty years of coding theory as a pursuit of limits Shannon proved reachable but never built.
- Shannon himself, in dissent — “The Bandwagon” (1956): the field's founder as its first and best-humored critic.
- A Mind at Play — Soni and Goodman's 2017 biography, the standard account of the man and the paper.