cost and latency optimization

A feature that is brilliant but too slow or too expensive never ships. Every LLM call costs money per token and takes time to produce each word, so as usage grows these two numbers decide whether your product is viable. Optimising them is the constant background engineering of any serious LLM application: getting the same quality for fewer tokens, fewer calls, or a smaller model.

The biggest levers are choosing the smallest model that passes your evals, trimming the prompt and retrieved context so you pay for fewer input tokens, and capping output length. Beyond that: cache repeated or similar requests, route easy queries to a cheap model and only hard ones to the flagship, batch requests on the server, stream tokens so the user sees output immediately even if the full answer is slow, and run shorter prompts in parallel. Many teams also distil or fine-tune a small model to replace expensive few-shot prompting once they have data.

These optimisations trade against quality and complexity, so they are guided by measurement, not guesswork — you cut tokens only as far as your evals still pass. Cost and latency also interact: streaming improves perceived speed without changing the bill, while a smaller model improves both at some quality risk. Getting this balance right is often what turns a promising prototype into a product that can scale to many users affordably.