The token: an anatomy

The unit of the AI economy, taken apart: definition, price formation, margin capture, bottlenecks, and the limits of compatibility. Completes the trilogy with Who serves the tokens? and The price of a token.

1. What a token fundamentally is

A token is not a word. It is a subword unit from a fixed vocabulary — typically 50,000–250,000 entries learned by byte-pair encoding: frequent strings get their own token, rare words shatter into pieces. In English, one token averages about three-quarters of a word; in French, code, or Swahili the ratio differs, because the vocabulary was optimized for a particular training mix. Three identities are fused in this one object. A unit of language: the quantized symbol through which a model perceives text — in Shannon's terms (1948), the alphabet of a source model of human language. A unit of compute: generating one token means one full forward pass — roughly two floating-point operations per model parameter — so a token is a fixed quantum of physical work, of FLOPs, joules and grams of CO₂. A unit of billing: the industry meters intelligence per million of them. The entire token economy rests on the fusion of these three — and, as we will see, on the fact that the fusion is vendor-specific.

2. How the price is decided

1. The physical floor. Cost per token = FLOPs per token × cost per FLOP. The first term scales with model size (hence small models are cheap); the second bundles GPU depreciation over ~4–5 years, power, cooling and networking. For a frontier model, the raw floor is fractions of a cent per thousand tokens — when the hardware is busy.
2. Utilization is the real variable. An accelerator idles at the same hourly cost as one running flat out. Batching many requests through each forward pass divides cost per token by the batch size; this is why serving economics reward scale, why latency-sensitive traffic costs more, and why “provisioned capacity” discounts exist.
3. Input ≠ output. Reading a prompt (prefill) processes all tokens in parallel and saturates compute; generating (decode) emits one token at a time and is bottlenecked by memory bandwidth. That asymmetry — not marketing — is why output tokens list at three to five times input, and why re-served context (cached input) is discounted up to 10×: the work was already done.
4. Above the floor, strategy. Actual prices are set on a competitive ladder — each vendor positions against the GPT/Claude/Gemini price-performance frontier — with frontier models priced on value, commodity tiers priced near cost, and share-buying tiers priced below it (much consumer serving, and some Chinese API pricing, is deliberate subsidy). Training costs, notably, are sunk: they justify the enterprise, not the per-token price.

3. Who gets the margin

Follow a euro paid for a million tokens down the stack. Electricity and land take single-digit percents. TSMC takes its foundry margin on the chip. Nvidia takes the fattest slice in the history of hardware — gross margins around 70–75% on accelerators that the whole stack must buy. The cloud (or the lab's own datacenter) layers 20–40% on raw compute. The model lab keeps what remains: healthy gross margin on API traffic in the good cases, negative in the subsidized ones — and always before subtracting the training bill, which is why labs in the middle of the stack burn capital while their suppliers print it. The application layer on top (agents, copilots, vertical software) resells tokens inside seats and outcomes at software margins. The pattern is the old one: in a gold rush, margin pools at the shovel-maker below and at the claim-owner above; the diggers in between fight over the rest. The exceptions prove it: Google escapes the squeeze by owning the whole column from TPU to consumer surface, and every lab's custom-silicon project (OpenAI with Broadcom, Anthropic on TPUs and Trainium) is an attempt to claw back the Nvidia slice.

4. Where the bottlenecks are

1. Power. The binding constraint of the late 2020s: gigawatt campuses queue years for grid interconnects, and nuclear PPAs (Microsoft–Constellation) exist because electrons, not chips, now gate buildouts.
2. HBM and advanced packaging. High-bandwidth memory (SK Hynix, Micron, Samsung) and TSMC's CoWoS capacity are the literal chokepoints of accelerator supply — sold out quarters ahead.
3. Leading-edge fabrication. One company (TSMC) at scale, fed by one lithography monopoly (ASML, with Zeiss optics inside) — the most concentrated dependency chain in the world economy, traceable on the company map.
4. Memory bandwidth at decode. Inside the box, tokens per second are limited by how fast weights and KV cache move, not by FLOPs — the physics behind output pricing and the speculative- decoding cottage industry.
5. Trust and evaluation. The soft bottleneck: the scarce resource in enterprise adoption is not tokens but confidence that an agent's tokens can be acted on — evals, audits and accountable deployment (the argument of the factory as institutional fact).

5. Compatibility — and fundamental non-compatibility

The market talks about tokens as if they were a commodity, like kilowatt-hours. The truth is stranger: the protocol has commoditized while the substance has not.

The deep point: a token is only meaningful relative to the model that emits it — its tokenizer, weights, caches and habits. Tokens are therefore fundamentally non-fungible across vendors, even when the API calls look identical. What routers and standards have built is compatibility of plumbing; what remains vendor-specific is everything the plumbing carries. This is why the token market behaves like the barbell described in The price of a token: perfectly substitutable at the commodity end where any model will do, and deeply locked-in at the frontier end where fine-tunes, evals, embeddings and trust have married one vendor's meter.