holistic evaluation

Holistic evaluation refuses to reduce a model to one number. Instead it reports many metrics across many scenarios at once — accuracy, but also calibration, robustness, fairness, bias, toxicity, efficiency, and cost — so you see the whole shape of a model's behaviour, including its trade-offs. The point is that a model topping the accuracy chart might also be the slowest, most biased, or least calibrated, and a single leaderboard number hides exactly the things that matter for deployment.

The best-known framework lays out a grid of scenarios crossed with metrics and runs every model through the same cells, then publishes the full table rather than a single rank. This makes evaluation transparent: you can see what was tested, under what conditions, and where each model is strong or weak, and choose a model for your need rather than for a headline.

The cost is complexity. More metrics mean more compute, harder interpretation, and decisions about how to weigh competing goods — is a 2-point accuracy gain worth twice the latency? Holistic evaluation does not answer that for you; it makes the trade-off visible so a human can decide honestly.