cost per token

At scale, the question that decides whether a feature ships is plain: what does one token cost? Providers quote prices per million tokens, usually splitting cheaper input (prefill) tokens from pricier output (decode) tokens, because the two phases stress the hardware in completely different ways.

The underlying cost is hardware time: the GPU's cost per hour divided by the tokens per second the system can sustain. So anything that raises sustained throughput — continuous batching, quantization, speculative decoding, bigger batches, prefix caching — directly lowers cost per token. Output tokens cost more because memory-bound decode produces them one at a time, while prefill chews through input tokens in parallel. This single number ties the whole serving stack to the business: it sets API prices, decides which use cases are economical, and is why so much engineering effort goes into squeezing more tokens out of every GPU.