JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Fast, Cheap, and Yours: Cost, Latency, Caching, and Smaller/Open Models

A feature users love can still die from its bill or its lag. Here is how to make LLM apps fast, affordable, and self-owned.

The cost and latency reality

Two numbers decide whether an LLM feature survives contact with real usage: how much each call costs and how long it takes. Cost scales with tokens — both the tokens you send and the tokens generated — so a chatty prompt or a giant retrieved context can quietly multiply your bill. Latency is dominated by time to first token and then generation speed. Cost and latency optimization is the craft of cutting both without cutting quality you can measure.

C = n_{\text{in}}\,p_{\text{in}} + n_{\text{out}}\,p_{\text{out}}

Total cost is input tokens times input price plus output tokens times output price — and output tokens usually cost more.

Optimization techniques

  1. Right-size the model per task: send easy requests to a small fast model and reserve the big expensive one for genuinely hard ones (a router, from guide 2).
  2. Trim the prompt: shorter system prompts, fewer few-shot examples, and tighter retrieved context all cut input tokens on every single call.
  3. Cap and stream output: set a sensible max-token limit, and stream so perceived latency drops even when total time does not.
  4. Batch and parallelize independent calls instead of waiting on them one by one.
Each output token is produced one at a time, so capping and streaming this generation loop is where latency and cost are won.

Diagram of an autoregressive loop emitting one token at a time and feeding each back as input.

Caching: don't pay twice

Caching LLM responses is the highest-leverage optimization because the cheapest call is the one you never make. Exact-match caching stores the answer for an identical request — perfect for repeated lookups. Semantic caching matches on meaning, returning a stored answer when a new question is close enough to a previous one. And prompt caching (a provider feature, related to prompt caching) reuses the cost of a long, stable prefix — like a big system prompt or shared context — across many calls. Each comes with a freshness tradeoff: never cache something that must reflect live data.

Right-sizing: small language models

Bigger is not always better. A small language model — a few billion parameters instead of hundreds — can match a giant on a narrow task once you fine-tune it on your data, while costing a fraction per call and running far faster. Two techniques get you there: distillation, where a big model's outputs train a small one to imitate it, and quantization, which shrinks the weights to lower precision so the model fits on cheaper hardware with little quality loss. The discipline is to right-size: use the smallest model that still passes your evals.

Scaling laws show the returns from a bigger model shrink — a fine-tuned small model can match a giant on a narrow task.

Log-log plot of loss falling as model scale grows, with diminishing returns.

On-device and the open ecosystem

Small enough models can run on the device itself — a phone or laptop — which means zero per-call cost, instant response, and data that never leaves the user. The tradeoff is a capability ceiling and the burden of shipping models to varied hardware. Behind all of this is the open-source LLM ecosystem: open-weight models you can download, fine-tune, and self-host with serving frameworks. Self-hosting buys you control, privacy, and predictable cost at scale — in exchange for owning the operations, GPUs, and uptime yourself.