When one agent is not enough
A single agent stuffing every tool, instruction, and intermediate result into one context eventually buckles: the prompt grows unwieldy, distinct skills interfere, and the model loses focus. The natural move is division of labor — a multi-agent system of specialists, each with a narrow role, a small toolset, and a clean context. A 'researcher' gathers sources, a 'coder' writes the patch, a 'critic' reviews it. Each stays sharp because each does one thing.
Orchestration patterns
Multi-agent orchestration is the discipline of coordinating these specialists: defining roles, routing messages, and managing handoffs so the team converges instead of thrashing. A few patterns recur. Supervisor–worker puts one orchestrator in charge of decomposing the task and delegating subgoals to workers, collecting their results — essentially the subgoal hierarchy from guide 3 with each subgoal assigned to a dedicated agent. Pipeline chains agents in a fixed sequence, each consuming the previous output. Debate / critique pairs a generator with an adversarial reviewer to catch errors before they ship.
Under the hood each agent is still running the reasoning–acting loop from guide 1 — but now one agent's output is another agent's observation. The orchestration layer is itself an environment, and the same control-loop disciplines apply: cap the rounds, detect non-progress, and make 'done' an explicit, checkable event rather than a vibe.
Evaluating agents end to end
Evaluating an agent is far harder than scoring a single answer, because an agent succeeds or fails over a trajectory of many steps and tool calls. Agent evaluation benchmarks therefore measure end-to-end task completion in a real or simulated environment: did the agent achieve the goal, not did any single utterance look good. The gold standard uses checkable outcomes. SWE-bench, for instance, hands the agent a real GitHub issue and runs the repository's own test suite on the patch it produces — success is the tests going green, an objective, contamination-resistant signal no judge needs to interpret.
When you cannot fully automate the check, report distributional metrics, not a single number: success rate over many tasks with a confidence interval, cost and step-count per task, and a breakdown of where trajectories fail. An agent that solves 60% of tasks at ten cents each is a very different system from one that solves 62% at five dollars each — and an aggregate accuracy hides that entirely.
Report the success rate as an estimate with a confidence interval, never a lone number — the interval narrows as you test over more tasks n.
The judge problem
For open-ended outputs with no test to run, a common shortcut is to use a strong model as the grader — LLM-as-a-judge. It scales cheaply and correlates surprisingly well with human preference, but it imports real biases. Judges show position bias (favoring whichever answer is shown first), verbosity bias (longer looks better), and self-preference (a model rating its own family's outputs higher). And because the agent and the judge are often the same model, a confidently wrong judge can rubber-stamp a confidently wrong agent.
The research frontier
Step back and the open problems of agentic AI come into focus. Reliability is the wall: per-step accuracy of 95% still collapses to a coin flip over a 14-step task, so long-horizon competence demands either far higher per-step accuracy or robust recovery from inevitable errors. Cost and latency scale with every thought, branch, and agent, making the test-time-compute trade-off an economic question, not just a technical one. Evaluation itself lags capability — we struggle to build benchmarks that are realistic, contamination-resistant, and not saturated within months.
Why reliability is the wall: end-to-end success is per-step accuracy raised to the trajectory length, so 95% per step is barely a coin flip over 14 steps.
The throughline of this whole track is a single shift in stance: stop treating the model as an answer machine and start treating it as a controller embedded in a loop with the world. Reasoning, planning, memory, tools, teams, and evaluation are all answers to the questions that stance forces — how to decide, how to remember, how to act, how to coordinate, and how to know it worked. Getting all five right, reliably and affordably, is the frontier the field is now climbing.