Pass@k: What the Metric Family Hides

Wed, 04 Nov 2026 09:00:00 +0800

Coverage finds a passing attempt; reliability asks whether every attempt succeeds

Key Takeaways

Pass@k gives a model k independent tries at each problem and counts the problem solved if any one try passes the tests. The score is the share of problems solved that way. Estimate it by drawing more than k samples and using the unbiased formula from the HumanEval paper. Extrapolating from a measured single-try rate, as if each try were a coin flip, undercounts it.
Pass@k at large k measures coverage: whether a correct answer exists somewhere among the tries, not whether you can find it. Unless tests or another checker pick the winner, pass@100 is the ceiling of a system you cannot build, not an accuracy you can ship.
The same score can hide three different problems. The tests may be too weak to catch wrong code: the stricter HumanEval+ suite cut scores by up to 28.9 per cent. The model may get lucky once and fail to repeat it: pass^k, the chance that all k tries pass, falls fast as k grows. And a retry that reads the previous error message is not an independent try, so the formula no longer applies.
A self-training loop needs four numbers, not one: success on a single deterministic try, coverage at large k, pass^k, and task completion under the retry budget the system really gets. Each answers a different question. Any one of them alone misleads.

TL;DR

Pass@k reports the fraction of problems for which at least one of k samples passes the tests, and the Codex estimator makes that number unbiased for any k up to the sample count. It says nothing about which sample you would have picked, how often a single run succeeds, or whether the tests are strong enough to reject wrong code. Pair it with greedy pass@1, pass^k and a harder test suite before drawing conclusions about a model or a self-training loop.

Self-Training Agents on Eigenform Continuous Learning Tool Tips

Pass@k: What the Metric Family Hides

Key Takeaways

TL;DR