coding benchmarks

Coding benchmarks ask the model to write real programs and then check whether the code actually runs and passes tests. This is one of the rare places where grading is objective: you do not need a human or a judge model — you just execute the candidate solution against a battery of unit tests and see if it produces the right outputs. HumanEval, the classic suite, gives the model a function signature and docstring and asks it to fill in the body; SWE-bench raises the bar to fixing genuine bugs in large open-source repositories.

The standard score is pass@k: the model is allowed k attempts per problem, and it counts as solved if any attempt passes all tests. pass@1 measures one-shot reliability; pass@10 measures whether a correct solution is anywhere in its range. Because the signal is execution, not surface similarity, these benchmarks correlate unusually well with real usefulness for developers.

The caveats are that passing the given tests is not the same as being correct, secure, or maintainable, and popular problems leak into training sets. A model can also overfit to puzzle-style functions while struggling with the messy, under-specified work of an actual codebase.