request batching
/ ree-KWEST BACH-ing /
Request batching is grouping several incoming requests and running them through the model together in one go, instead of one by one. Recall that accelerator chips are like a kitchen of thousands of cooks: handing them a single order wastes most of the cooks. Batching is the maître d' who gathers a dozen orders before sending them all to the kitchen at once, so every cook is busy. The chip does roughly the same amount of work whether it processes one input or thirty-two, so batching is nearly free extra capacity.
The simplest form waits a few milliseconds to collect a batch, then runs it — trading a tiny bit of each user's waiting time for a large gain in total capacity. Cleverer schemes used in modern text generation, called continuous or dynamic batching, don't make finished requests wait for slow ones: as soon as one user's answer is done, that slot is freed and a new request slips in, keeping the chip packed without making anyone wait for the whole group. This is one of the single biggest levers for cutting the cost of running large language models.
Why it matters: batching is often the difference between an affordable service and a ruinously expensive one, sometimes improving cost-efficiency several-fold. The honest tradeoff is that it always adds some latency for individual requests, and it only helps when there is enough traffic to fill batches — at 3 a.m. with one user online, there is nothing to batch, and you pay full price per request. It is a throughput optimization, deliberately spending latency to buy capacity.
A server waits up to 10 ms after the first request arrives, sweeping up any others in that window. If 16 arrive, all 16 run through the GPU in a single pass — costing about the same time as one would alone, but answering 16 users for the price of one.
A 10 ms wait turns 16 separate jobs into one — the core trick behind cheap inference.
Batching is not a quality improvement — it changes nothing about the model's answers, only how efficiently the hardware produces them. It is purely an economics-and-speed lever, and it can only be pulled when enough requests are arriving to fill a batch.