caching LLM responses

If two users ask the same question, why pay the model to answer twice? Caching stores past results and serves them again when the same — or a similar — request comes in, cutting both cost and latency to near zero on a cache hit. It is one of the oldest tricks in computing applied to an expensive new component.

The simplest form is exact-match caching: hash the full prompt and reuse the stored completion if you have seen it before. Semantic caching goes further — it embeds the new query and returns a cached answer when an earlier query is close enough in meaning, catching paraphrases an exact match would miss. Separately, providers offer prompt caching that stores the processed form of a long, repeated prefix — a big system prompt or document — so you are not recharged full price to reprocess it on every call. Each needs a policy for when entries expire.

Caching shines when requests repeat and the right answer is stable, and it is one of the cheapest wins available. But it is dangerous when answers must be fresh, personalised, or time-sensitive — a stale cached reply can be worse than a slow correct one — and semantic caching can return a near-match that is subtly wrong for this particular query. The art is caching aggressively where outputs are reusable and not at all where they must be live.