pass@k

For code generation, a model rarely nails a hard problem on the first try but often succeeds if allowed several attempts. pass@k measures exactly that: the probability that at least one of k independently sampled solutions passes all the unit tests. pass@1 is single-shot reliability, the number that matters when you run a model once; pass@100 reveals whether the right answer is anywhere in the model's reachable distribution.

Naively estimating pass@k by drawing exactly k samples is high-variance. The standard HumanEval estimator instead draws a larger n (with n greater than or equal to k) samples per problem, counts the c that pass, and computes an unbiased pass@k in closed form via the complementary hypergeometric probability of drawing zero correct samples among k. This gives stable estimates and lets you read off the whole pass@k curve from a single sampling run.

pass@k with large k flatters models good at exploration but unreliable per attempt, and it requires an oracle — the test suite — so it only measures what the tests cover and misses silent correctness gaps. Sampling temperature matters too: higher temperature aids large-k coverage but hurts pass@1, so the curve, not one point, tells the real story.