When Pass@1 Craters, Your Agent Might be Learning to Learn

Mon, 26 Oct 2026 00:00:00 +0800

A first attempt that fails and maps the container can be worth more than a lucky first success

Key Takeaways

State of the art: Pass@1, the share of tasks solved on the first sampled attempt, is a standard measure of coding ability, and in a self-training loop a falling pass@1 is usually read as forgetting, mode collapse or reward hacking. That reading is sound for static, single-shot benchmarks.
Unresolved problem: Training on whole episodes means a cheap failing first attempt is learned together with the fix its error message guided; whether step-level credit assignment would suppress this probing is still open, and evaluation needs metrics that count what each attempt learns." for “It seemed like the models had learned to use the debug cycle as an extra exploration channel. Deliberately writing failing code on their first try could elicit error messages that would provide useful information to improve performance later on.” for “Why does pass@1 degrade in self-training loops, and is it always catastrophic forgetting? In our disk-space self-training runs, first-attempt success fell to zero in two of three lineages while more of the agents’ code ran and their HumanEval score stayed close to the base model’s, it appeared that the agents had learned to submit a failing first attempt, read the error and get higher performance on later attempts as a result. Below we show the divergence, the mechanisms that let this strategy survive SFT on whole successful episodes, and how to check for it in your own loop.
Research direction: We are investigating periodic SFT on whole episodes the environment accepted (disk space really freed), trained into fresh LoRA adapters with no exploration bonus. Keeping whole episodes means a cheap failing first attempt is learned together with the fix its error message guided; whether step-level credit assignment would suppress this probing is still open, and evaluation needs metrics that count what each attempt learns.

TL;DR

Why does pass@1 degrade in self-training loops, and is it always catastrophic forgetting? In our disk-space self-training runs, first-attempt success fell to zero in two of three lineages while more of the agents’ code ran and their HumanEval score stayed close to the base model’s, because the agents had learned to submit a cheap first attempt, read the error and fix it on the next try. Below we show the divergence, the mechanisms that let this strategy survive SFT on whole successful episodes, and how to check for it in your own loop.

Why First-Attempt Pass@1 Drops as SFT Agents Improve

Wed, 21 Oct 2026 12:18:00 +0800

When retries are free, the first attempt becomes a probe

Key Takeaways

State of the art: Since Codex, pass@1 has been the standard proxy for coding competence, and self-training methods such as STaR and ReST predict that SFT on filtered successes raises first-attempt success together with overall success.
Unresolved problem: For agents that can retry, first-attempt correctness and multi-step competence can move in opposite directions. In our self-training runs (Survival is the Only Reward), pass@1 fell to zero by generation 6 in two of three lineages while the agents wrote more working code: they had learned to submit a probe, read the error, and solve on the next attempt.
Research direction: We are investigating periodic SFT on environment-validated episodes with a sliding window LoRA, with no RL objective and no exploration bonus. Because the whole surviving episode enters the buffer, the probe and the error-conditioned repair are learned together, so filtering to first-attempt successes would likely raise pass@1 and destroy the strategy; evaluation needs metrics that count attempts and the information each one gains.

TL;DR

Why would pass@1 drop while an LLM agent gets better at its task? In our self-training sandbox, agents fine-tuned with SFT on their own successful episodes learned to treat the first submission as a probe: it fails, the compiler or shell error narrows the next attempt, and the episode succeeds, so pass@1 dropped to zero or near it while the agents wrote more working code. Below we show the divergence, annotated trajectories, three mechanisms that explain it, and how to detect the same signature in your own training loop.

Evaluation on Eigenform Continuous Learning Tool Tips

When Pass@1 Craters, Your Agent Might be Learning to Learn

Key Takeaways

TL;DR

Why First-Attempt Pass@1 Drops as SFT Agents Improve

Key Takeaways

TL;DR