model serving and inference
/ MOD-ul SUR-ving and IN-fur-uns /
Model serving is everything that happens after a model is trained: putting it somewhere it can answer real requests, reliably and fast, for real users. Inference is the act of actually using the model — feeding it an input and getting a prediction back. Training is like writing and rehearsing a play for months; serving is opening night, every night, with a live audience who expect the show to start on time and never crash.
Concretely, serving wraps the model in a piece of software that listens for requests, runs the input through the network, and returns the answer — a translation, a recommendation, a generated paragraph. Around that sits the unglamorous but essential machinery: load balancing across many copies of the model, scaling up when traffic spikes, falling back gracefully when something breaks, logging what was asked, and keeping latency low. The model is the engine; serving is the whole car around it that makes the engine useful on a real road.
Why it matters: a model that wins benchmarks but can't be served affordably and reliably is a science project, not a product. Most of the real cost and engineering pain of AI in industry lives here, not in training. The candid truth often surprises newcomers: training happens once, but inference runs millions of times a day forever, so over a model's lifetime serving usually dominates the total bill — and a model's accuracy means nothing to a user staring at a spinning loading icon.
You type a question into a chatbot. Your text travels to a server, where a copy of the model runs inference and streams words back. Behind the scenes, thousands of other users are hitting the same fleet of servers at the same instant — serving is what keeps each request answered without the whole thing falling over.
One user's question, a whole invisible fleet keeping every question answered.
A common mistake is treating serving as an afterthought once the model 'works.' In practice, decisions about latency, cost, and scale shape which models are even viable — sometimes a slightly less accurate model that serves cheaply beats a perfect one nobody can afford to run.