When Pass@1 Craters, Your Agent Might be Learning to Learn

Mon, 26 Oct 2026 00:00:00 +0800

A first attempt that fails and maps the container can be worth more than a lucky first success

Key Takeaways

State of the art: Pass@1, the share of tasks solved on the first sampled attempt, is a standard measure of coding ability, and in a self-training loop a falling pass@1 is usually read as forgetting, mode collapse or reward hacking. That reading is sound for static, single-shot benchmarks.
Unresolved problem: Training on whole episodes means a cheap failing first attempt is learned together with the fix its error message guided; whether step-level credit assignment would suppress this probing is still open, and evaluation needs metrics that count what each attempt learns." for “It seemed like the models had learned to use the debug cycle as an extra exploration channel. Deliberately writing failing code on their first try could elicit error messages that would provide useful information to improve performance later on.” for “Why does pass@1 degrade in self-training loops, and is it always catastrophic forgetting? In our disk-space self-training runs, first-attempt success fell to zero in two of three lineages while more of the agents’ code ran and their HumanEval score stayed close to the base model’s, it appeared that the agents had learned to submit a failing first attempt, read the error and get higher performance on later attempts as a result. Below we show the divergence, the mechanisms that let this strategy survive SFT on whole successful episodes, and how to check for it in your own loop.
Research direction: We are investigating periodic SFT on whole episodes the environment accepted (disk space really freed), trained into fresh LoRA adapters with no exploration bonus. Keeping whole episodes means a cheap failing first attempt is learned together with the fix its error message guided; whether step-level credit assignment would suppress this probing is still open, and evaluation needs metrics that count what each attempt learns.

TL;DR

Why does pass@1 degrade in self-training loops, and is it always catastrophic forgetting? In our disk-space self-training runs, first-attempt success fell to zero in two of three lineages while more of the agents’ code ran and their HumanEval score stayed close to the base model’s, because the agents had learned to submit a cheap first attempt, read the error and fix it on the next try. Below we show the divergence, the mechanisms that let this strategy survive SFT on whole successful episodes, and how to check for it in your own loop.

Meta-Learning on Eigenform Continuous Learning Tool Tips

When Pass@1 Craters, Your Agent Might be Learning to Learn

Key Takeaways

TL;DR