tokens and the cost of inference
/ TOH-kunz and the kawst uv IN-fur-uns /
Every time you use a language model — "inference," as opposed to the one-time "training" — it costs real money and energy, and the unit that meter runs on is the token. A token is a chunk of text the model reads and writes in: roughly a short word or a word-fragment ("banana" might be one token, "unbelievable" two or three). The model doesn't see letters or words exactly; it sees a stream of tokens, and almost every API charges per token, counting both what you send in and what the model writes out.
Two facts make this practically important. First, you pay for the whole context every time: if you carry a long conversation or paste a big document, all of it is re-read on each new turn, so a 50-page report fed in repeatedly racks up cost fast. Second, generating output is inherently sequential — the model produces one token at a time, each depending on the last — so a long answer is slower and pricier than a short one, and there is no way to simply parallelize it away. Cost scales with how much text goes in and comes out.
Why this matters beyond the bill: it shapes how these systems are designed and what they cost society. A single query is cheap, but a popular service handling billions of them spends heavily on electricity and specialized chips, with a real environmental footprint. It is also why techniques like distillation, quantization, and caching exist — all are ways to wring more answers out of each token's worth of computation. The token is the quiet economic atom of the whole industry.
You paste a 10,000-token contract and ask five follow-up questions in one chat. Because the whole contract is re-read on each turn, you're billed for roughly 50,000 input tokens, not 10,000 — plus the output tokens for each answer. Trimming the document to only the relevant clauses can cut the bill by most of that.
The whole context is re-read — and re-billed — on every single turn.
A common surprise: cost grows with the whole conversation, not just your latest message, because the full context is reprocessed every turn. And because output is generated one token at a time, you cannot make a long answer fast simply by adding more hardware — length itself is the bottleneck.