Collapsing Reward, Optimisation, and Persistence in Agents
Collapsing reward, optimisation, and persistence into physical survival embraces reward hacking: any method that gets the reward is a valid adaptation.
All
AI 16
Continuous Learning 15
LoRA 13
Fine-Tuning 11
Self-Training 8
SFT 7
Catastrophic Forgetting 6
Continual Learning 6
AI Agents 5
Data Selection 3
Self-Training
The Katalin Trap: When Cherry-Picking Training Data Fails
Cherry-picking training data for peak performance can trap agents in dead environments, while sliding window buffers help prevent self-training collapse.
Variance Deletion: Why Cherry-Picking Data Collapses Agents
Cherry-picking training data for peak performance deletes behavioural variance and collapses agents; a sliding window buffer preserves generalisation.
When Pass@1 Craters, Your Agent Might be Learning to Learn
Pass@1 fell to zero in our self-training loop while the agents improved: they had learned to use failing first attempts as probes. Why, and what to measure.
Why Cherry-Picking Your Best Data Can Kill Generalisation
Data selection for fine-tuning in a self-training loop: training on the best data collapsed in our 13-generation study, while recent data kept improving.
Sliding Window SFT vs Best-of-N in Continual Learning
In our 13-generation continual learning study, top-3 historical generation curation collapsed while a K=3 sliding window SFT buffer adapted on 4x less data.
Why First-Attempt Pass@1 Drops as SFT Agents Improve
Why can pass@1 fall to zero while an LLM agent gets better at its task? Our 13-generation SFT study, the probe-then-solve mechanism, and how to detect it.
Sliding Window SFT: Continual Learning Without Data Scaling
In a 13-generation agent study, a K=3 sliding window SFT loop retains 52.6% of full-history offline composite score and 58.4% of MB freed on 25% of the rows.