JOVANA
Explore Library Glossary Getting Started Three Levels Fields How it works Mission
Join the mission
All guides

Shipping It: Context Extension, Prompt Caching, and Model Merging

A great model that is too slow or too expensive never ships. Stretch the context window, cache the prompt prefixes you reuse, and merge fine-tuned checkpoints without retraining — the engineering that turns a checkpoint into a product.

The serving mindset

Training optimizes a loss; serving optimizes latency, throughput, and cost per token. The same checkpoint that scores beautifully on a benchmark may be unusable in production if the first token takes three seconds or each request costs a nickel. Inference cost dominates the lifetime budget of a deployed model — you train once but serve billions of times — so the engineering in this guide is where most of the money is actually saved.

The central data structure of LLM serving is the KV cache: the keys and values each attention layer computed for every token so far, stored so you do not recompute them on the next step. Almost every serving optimization is, at heart, a way to build the KV cache less often or store it more cheaply. Keep that lens and the three techniques below click into place.

Autoregressive serving emits one token at a time; the KV cache stores each layer's keys and values so the next step never recomputes the prefix.

A loop feeding each generated token back as input to produce the next token.

Context window extension

A model pretrained with a 4k-token context window cannot, out of the box, handle a 100k-token document — feed it more than it saw in training and quality falls off a cliff. The bottleneck is positional encoding: the model only learned to interpret position signals up to its training length. Context window extension is the set of light-touch techniques that push that limit out, usually with a short fine-tune rather than full retraining.

Most modern models use rotary position embeddings (RoPE), which encode position as rotations at a spectrum of frequencies — and that structure is exactly what makes extension cheap. The simplest move, position interpolation, squeezes the new longer positions back into the range the model already understands. YaRN refines this: rather than scaling every frequency uniformly, it interpolates the high frequencies (which carry local detail) gently and the low frequencies (which carry long-range position) more aggressively, getting a 4×–16× extension from a tiny fine-tune with little quality loss.

m \;\longmapsto\; m\cdot\dfrac{L_{\text{train}}}{L_{\text{target}}},\qquad L_{\text{target}}>L_{\text{train}}

Position interpolation rescales every RoPE position index back into the range the model saw in training, so a longer context no longer falls off the cliff.

Prompt caching

Many requests share a long, identical prefix: the same multi-thousand-token system prompt, the same few-shot examples, the same RAG instructions, the same tool schemas. Without caching, every request recomputes the KV cache for that whole shared prefix from scratch — paying for the same thousands of tokens over and over. Prompt caching computes the prefix's KV states once and reuses them across requests, so only the genuinely new suffix tokens are processed.

The payoff is large because the prefill phase (processing the input prompt) is compute-bound and often dominates latency for long prompts. Caching turns a cold request that processes 8,000 prompt tokens into a warm one that processes only the 50 new tokens — slashing both time-to-first-token and cost. Serving engines implement this as prefix caching, hashing prompt prefixes and storing their KV blocks for reuse; you design your prompts to maximize the shared, stable prefix and put the variable parts at the end.

Model merging

Suppose you fine-tuned three specialists from the same base — one for code, one for math, one for chat. Serving three models is expensive and routing between them is fiddly. Model merging combines several fine-tuned checkpoints in weight space into a single model, with no additional training and no GPUs spent on gradients — you just do arithmetic on the weights.

The simplest recipe averages the weights. A sharper one, task arithmetic, computes each specialist's task vector — the difference between its weights and the base model's — and adds the task vectors you want, letting you literally compose skills. More careful methods (TIES, DARE) prune and resolve sign conflicts between task vectors so they interfere less. It works because fine-tunes from a shared base tend to stay in a connected, low-loss region of weight space.

\tau_i=\theta_i^{\text{ft}}-\theta_0,\qquad \theta_{\text{merge}}=\theta_0+\sum_i \lambda_i\,\tau_i

Task arithmetic forms each specialist's task vector as its weight delta from the base, then adds the scaled vectors to merge skills without any retraining.

From checkpoint to product

Step back and see the full arc of this track. You took a base model, gave it behavior with SFT, adapted it cheaply with LoRA-family PEFT, aligned it with preference optimization, grounded it with RAFT and locked its format with structured decoding, and finally made it fast and affordable with context extension, prompt caching, and merging. That end-to-end pipeline — data, adaptation, alignment, grounding, serving — is LLM engineering.