Pass@k: What the Metric Family Hides
Pass@k measures coverage, not reliability. How to estimate it without bias, why HumanEval+ and pass^k reorder models, and what to report for agent loops.

Pass@k measures coverage, not reliability. How to estimate it without bias, why HumanEval+ and pass^k reorder models, and what to report for agent loops.

Pass@1 fell to zero in our self-training loop while the agents improved: they had learned to use failing first attempts as probes. Why, and what to measure.

Why can pass@1 fall to zero while an LLM agent gets better at its task? Our 13-generation SFT study, the probe-then-solve mechanism, and how to detect it.