SWE-bench

SWE-bench raises the bar from write one function to fix a real bug in a real repository. Each task is a genuine GitHub issue from a popular Python project paired with the codebase at that moment; the agent must navigate many files, edit the right ones, and produce a patch. Success is decided by running the project's own test suite — including the tests the human fix originally added — so passing means the change actually resolves the issue.

An instance bundles the issue text, the repository snapshot, and two hidden test sets: fail-to-pass tests (broken before, fixed by a correct patch) and pass-to-pass tests (must stay green, catching regressions). The model's generated diff is applied and the suite is executed in a sandboxed environment; the score is the percentage of issues fully resolved. SWE-bench Verified is a human-validated 500-instance subset that removes underspecified or impossible problems, and SWE-bench-Lite is a cheaper subset — harness and subset choice dominate comparability.

Scores depend heavily on the agent scaffold (retrieval, tools, iteration budget), not the base model alone, so a SWE-bench number describes a whole system, not a model. The original set has documented contamination and specification issues — the motivation for Verified — and resolving a test-gated issue is not proof of clean, maintainable code that a human reviewer would accept.