When Pass@1 Craters, Your Agent Might be Learning to Learn
Pass@1 fell to zero in our self-training loop while the agents improved: they had learned to use failing first attempts as probes. Why, and what to measure.
All
AI 16
Continuous Learning 15
LoRA 13
Fine-Tuning 11
Self-Training 8
SFT 7
Catastrophic Forgetting 6
Continual Learning 6
AI Agents 5
Data Selection 3
Evaluation
Why First-Attempt Pass@1 Drops as SFT Agents Improve
Why can pass@1 fall to zero while an LLM agent gets better at its task? Our 13-generation SFT study, the probe-then-solve mechanism, and how to detect it.