agent evaluation benchmarks

Agent evaluation benchmarks measure what a whole agent can actually accomplish, not just whether a model emits good text. They pose tasks with a definite outcome — fix this bug, complete this purchase, answer this multi-tool question — and score whether the agent reached the goal end to end. The shift is from grading an answer to grading an achievement, in an environment the agent must interact with over many steps.

Representative suites target different settings: SWE-bench for resolving real GitHub issues judged by passing tests, WebArena and its successors for navigating realistic websites, OSWorld for desktop computer-use tasks, GAIA for multi-step tool-using questions, and tau-bench for tool-and-rule-following in customer-service scenarios. The dominant metric is task success rate, often reported as pass@k with attempts allowed, alongside cost, number of steps, and safety violations. Good benchmarks emphasize execution-based, checkable outcomes over surface similarity to a reference.

Agent evaluation is genuinely hard: long-horizon tasks have many valid solution paths, environments are stochastic and can drift, and partial credit is awkward to define. Results are sensitive to the scaffold, the tools provided, and the prompt, so a headline success rate says as much about the harness as the model, and contamination or overfitting to a popular benchmark can inflate numbers without real capability gains.