SFT on Eigenform Continuous Learning Tool Tips

Collapsing Reward, Optimisation, and Persistence in Agents

Thu, 29 Oct 2026 14:20:00 +0900

When physical survival is the only reward, every working exploit is a legitimate adaptation

Key Takeaways

Embrace reward hacking as valid adaptation: Conventional AI training treats unexpected shortcuts as “reward hacking” only because a designer has a preferred method in mind. When reward, optimisation, and persistence collapse into one physical outcome, no method that gets the reward is a bad method.
Survival makes the environment the only referee: In our Negative Space Learning experiments, agents inside Linux containers persist into future training buffers whenever they free real disk space. Unconventional shortcuts, lateral system surgery, and emergent exploits that reclaim real bytes directly buy the lineage’s survival.
Aggressive exploitation produces superior training data: In our 4×4 crossed fine-tune at W5, the buffer generated by ruthless confidence-weighted exploitation (DFT) trained to 98.8% under SFT and 96.8% under ASFT, proving that unpoliced reward-seeking harvests extraordinarily strong trajectories.
Stay plastic enough to find the next exploit: An opportunistic agent only fails when it stops adapting to new environments. Anchoring confidence-weighted updates to the base model (ASFT, 84.5% at M10; 100% when scheduled DFT -> ASFT) and curating memory by recency (Miri, 3.322) rather than past peak scores (Katalin, -0.008) keeps the lineage discovering fresh exploits round after round.

TL;DR

Most reinforcement learning pipelines treat reward hacking as an enemy to be engineered away with stricter rubrics and penalty terms. Collapsing reward, optimisation, and persistence into a single physical substrate takes the exact opposite view: it embraces reward hacking, operating on the principle that no method that gets the reward is a bad method. In Negative Space Learning, freeing disk space inside Linux containers is both the reward signal and the physical storage budget that keeps an agent’s training history alive. Any shortcut or exploit that reclaims real storage is a legitimate evolutionary adaptation. Once you welcome every working exploit, the job of the training loop is no longer to police the agent’s behaviour, but to keep the agent adaptable: pairing KL-anchored fine-tuning (ASFT) with sliding window training lets lineages cash in on aggressive exploits today while staying flexible enough to invent new ones tomorrow.

When Pass@1 Craters, Your Agent Might be Learning to Learn

Mon, 26 Oct 2026 00:00:00 +0800

A first attempt that fails and maps the container can be worth more than a lucky first success

Key Takeaways

State of the art: Pass@1, the share of tasks solved on the first sampled attempt, is a standard measure of coding ability, and in a self-training loop a falling pass@1 is usually read as forgetting, mode collapse or reward hacking. That reading is sound for static, single-shot benchmarks.
Unresolved problem: Training on whole episodes means a cheap failing first attempt is learned together with the fix its error message guided; whether step-level credit assignment would suppress this probing is still open, and evaluation needs metrics that count what each attempt learns." for “It seemed like the models had learned to use the debug cycle as an extra exploration channel. Deliberately writing failing code on their first try could elicit error messages that would provide useful information to improve performance later on.” for “Why does pass@1 degrade in self-training loops, and is it always catastrophic forgetting? In our disk-space self-training runs, first-attempt success fell to zero in two of three lineages while more of the agents’ code ran and their HumanEval score stayed close to the base model’s, it appeared that the agents had learned to submit a failing first attempt, read the error and get higher performance on later attempts as a result. Below we show the divergence, the mechanisms that let this strategy survive SFT on whole successful episodes, and how to check for it in your own loop.
Research direction: We are investigating periodic SFT on whole episodes the environment accepted (disk space really freed), trained into fresh LoRA adapters with no exploration bonus. Keeping whole episodes means a cheap failing first attempt is learned together with the fix its error message guided; whether step-level credit assignment would suppress this probing is still open, and evaluation needs metrics that count what each attempt learns.

TL;DR

Why does pass@1 degrade in self-training loops, and is it always catastrophic forgetting? In our disk-space self-training runs, first-attempt success fell to zero in two of three lineages while more of the agents’ code ran and their HumanEval score stayed close to the base model’s, because the agents had learned to submit a cheap first attempt, read the error and fix it on the next try. Below we show the divergence, the mechanisms that let this strategy survive SFT on whole successful episodes, and how to check for it in your own loop.

Why Cherry-Picking Your Best Data Can Kill Generalisation

Mon, 26 Oct 2026 00:00:00 +0800

Keeping only the best data trains a model for a world that has already moved on

Key Takeaways

State of the art: Self-training methods such as STaR and ReST generate data, keep the samples that score best, and train on them. Filtering for quality is the default, and it works well when the task distribution stays fixed.
Unresolved problem: When the agent’s own actions change its environment, the best data describes a state that no longer exists. We tested this via a self-directed exploration and training experiment (Survival is the Only Reward), the lineage trained on its three highest-scoring generations reached 13.2% of space freed by v5 and fell back to its starting 4.5% by v13, while the lineage trained on its three most recent generations, with the same data budget, ended at 12.0% and still rising.
Research direction: Why would an agent trained only on the best data collapse? Is it the data itself, the objective or some combination of both. Join us for a deep dive.

TL;DR

In our 13-generation runs on a disk-cleanup task, an agent trained on its three best generations peaked early and then fell back to where it started, while one trained on its three most recent generations, with the same data budget, kept improving: in an environment the agent itself depletes, the best data goes stale first. Below we argue that the window acts as a regulariser that keeps training data close to what the agent does now, compare it with a KL anchor at the loss level, and suggest experiments to test the idea.

Continual Learning Under Self-Induced Distribution Shift

Fri, 23 Oct 2026 15:00:00 +0800

When an agent’s actions permanently alter its environment, keeping past data preserves contradictions; a sliding window tracks the new reality

Key Takeaways

The Exogenous Shift Fallacy: Standard continual learning assumes the environment shifts externally while task distributions remain passive. In open-ended agent interaction, successful actions permanently mutate the environment, making the distribution shift endogenous and self-induced.
Catastrophic Remembering: Accumulating all historical trajectories in a cumulative replay buffer (Terese) causes policy stagnation not from forgetting, but from averaging over mutually incompatible past states from destroyed worlds.
The Sliding Window Solution: Retaining only the last three generations (\(K=3\)) of environment-validated trajectories (Miri) sustains monotonic improvement on a quarter of the data, achieving over 50% of peak performance without plateauing or collapsing.
Epistemic Hygiene via the Environment: A sliding window does not destroy knowledge; it enforces epistemic hygiene. Valid strategies are naturally re-derived and re-enter the window through live environmental feedback, while obsolete habits age out cleanly.

TL;DR

When an AI agent’s actions permanently alter its environment, standard continual learning assumptions collapse. Cumulative replay buffers hoard trajectories from worlds the agent itself destroyed, causing catastrophic averaging over contradictory states. Testing identical Qwen2.5-7B-Instruct agents in procedural Linux containers reveals that accumulating all history (Terese) plateaus at 65,000 rows, while a \(K=3\) sliding window SFT buffer (Miri) discards ghost states and achieves continuous, monotonic improvement on 4x less data by letting the live environment re-derive valid strategies.

Sliding Window SFT vs Best-of-N in Continual Learning

Wed, 21 Oct 2026 20:23:00 +0800

Recency beats peak quality when the agent’s own actions change its environment

Key Takeaways

State of the art: Best-of-n curation (rank sampled trajectories by reward, keep the winners, fine-tune on them) is a standard way to build high-quality SFT data, and continual learning methods such as EWC, synaptic intelligence, and experience replay in a static replay buffer are built to prevent forgetting.
Unresolved problem: When an agent’s own successes change its environment, the best past trajectories describe states that no longer exist. In our 13-generation study (Survival is the Only Reward), keeping the top three past generations (Katalin) peaked at v5 and fell to 6,349 MB freed per run at v13, below where it started, while keeping all past data (Terese) plateaued after v7.
Research direction: We are investigating a sliding window instead: each generation trains a fresh LoRA adapter on only the last three generations of environment-validated trajectories (Miri, about 15,000 rows), so a strategy stays in the buffer only while the current agent still produces it. In this single-lineage run, Miri kept improving to 16,223 MB freed per run at v13 and reached about half of Terese’s offline composite score (2.049 vs 3.898) on a quarter of the data.

TL;DR

Why does best-of-n curation collapse when an agent keeps training on its own data? In our three-lineage study on Qwen2.5-7B-Instruct, the agent’s own successes changed the containers it worked in, so the top-scoring past generations (Katalin) described a world that no longer existed and performance fell back to its starting level, while accumulating everything (Terese) plateaued. A sliding window of the last three generations (Miri) kept improving on a quarter of Terese’s data, because a strategy stays in the buffer only as long as the current agent still regenerates it.

Why First-Attempt Pass@1 Drops as SFT Agents Improve

Wed, 21 Oct 2026 12:18:00 +0800

When retries are free, the first attempt becomes a probe

Key Takeaways

State of the art: Since Codex, pass@1 has been the standard proxy for coding competence, and self-training methods such as STaR and ReST predict that SFT on filtered successes raises first-attempt success together with overall success.
Unresolved problem: For agents that can retry, first-attempt correctness and multi-step competence can move in opposite directions. In our self-training runs (Survival is the Only Reward), pass@1 fell to zero by generation 6 in two of three lineages while the agents wrote more working code: they had learned to submit a probe, read the error, and solve on the next attempt.
Research direction: We are investigating periodic SFT on environment-validated episodes with a sliding window LoRA, with no RL objective and no exploration bonus. Because the whole surviving episode enters the buffer, the probe and the error-conditioned repair are learned together, so filtering to first-attempt successes would likely raise pass@1 and destroy the strategy; evaluation needs metrics that count attempts and the information each one gains.

TL;DR

Why would pass@1 drop while an LLM agent gets better at its task? In our self-training sandbox, agents fine-tuned with SFT on their own successful episodes learned to treat the first submission as a probe: it fails, the compiler or shell error narrows the next attempt, and the episode succeeds, so pass@1 dropped to zero or near it while the agents wrote more working code. Below we show the divergence, annotated trajectories, three mechanisms that explain it, and how to detect the same signature in your own training loop.

Sliding Window SFT: Continual Learning Without Data Scaling

Wed, 21 Oct 2026 00:00:00 +0800

A sliding window retrains on recent successful trajectories, bounding data growth without performance collapse

Key Takeaways

State of the art: Continual learning methods typically treat forgetting as damage to prevent through full history accumulation, experience replay buffers, or weight-importance penalties, assuming that any training example dropped from the buffer is permanently lost to the policy.
Unresolved problem: In iterative agent self-training, accumulating every successful trajectory scales the training buffer linearly with diminishing returns per row, whereas curating the three historically highest-scoring generations (Katalin in Survival is the Only Reward) fragments strategy space and collapses at late generations (v12 to v13).
When working on self-retraining using sliding window data LoRAs that finetune agents on different segments of their own prior experience, we created three experimental model lineages: one trained on all of the agent’s prior experiments, both good and bad (Terese), one on 3000 lines of data from the best three runs (Katalin) and on on 3000 lines from the most recent three runs (Miri).
Candidate mechanism and scope: In an active environment such as our Negative Space Learning (NSL) sandbox, useful skills are re-executed by the current policy and re-enter the sliding window as fresh trajectories. While this was a limited test, it suggests a mechanism similar to the way in which humans retain the skills they use most, by simply retraining repeatedly on the most recently used capabilities.

TL;DR

When fine-tuning an LLM coding agent across multiple generations on its own successful trajectories, you must choose which historical datasets feed each update: accumulate all past generations (as we tried in our Terese lineage), select the three historically highest-scoring generations (our Katalin lineage), or keep a sliding window of the last three generations (our Miri lineage). In a 13-generation case study on Qwen2.5-7B-Instruct (Survival is the Only Reward), peak-score curation collapsed at v12 to v13, whereas the \(K=3\) recency window improved steadily without replaying older agent trajectories, reaching 14,938 MB freed per held-out run (58.4% of full accumulation’s 25,570 MB) and 2.049 offline cumulative composite score (52.6% of full accumulation’s 3.898, or 2.1× the composite-score units per training row) on a capped 15,000-row buffer.