Collapsing Reward, Optimisation, and Persistence in Agents

Q: Why did pure DFT crash from 100% to 0% at generation M6?

Unanchored Direct Fine-Tuning (DFT) weights token gradients by current model probability ($w(p)=p$), amplifying high-confidence actions without a base-policy tether. By M5 it reached 100% on the sealed panel, yet drifted 0.46–0.71 nats in forward KL, narrowing its repertoire until shifting M6 containers left it with zero bytes freed.

Q: What did the 4×4 crossed fine-tune prove about opportunistic exploits?

Training fresh single-step adapters on the frozen W5 buffer written by DFT yielded 98.8% under SFT and 96.8% under ASFT, the highest scores in the study. That result proves aggressive reward-seeking harvested strong, reusable exploits; DFT’s M6 crash came from unanchored over-concentration, not poisoned training data.

Q: How does ASFT preserve plasticity while amplifying winning trajectories?

Anchored Supervised Fine-Tuning (ASFT) combines DFT’s probability-weighted token updates ($w(p)=p$) with a forward-KL divergence penalty against the frozen base policy ($D_{\text{KL}}(\pi_0 \parallel \pi_\theta)$). The confidence weight locks onto decisive commands while the KL anchor preserves alternative strategies, reaching 84.5% at M10 (100% when scheduled DFT -> ASFT).

Collapsing reward, optimisation, and persistence in agents as an etched parchment engraving: three conduits of reward, optimisation, and persistence converging into a central brass physical byte reservoir, flanked by an isolated container bell-jar with drainage siphons and opportunistic bypass pipes — When physical survival is the only reward, every working exploit is a legitimate adaptation

TL;DR: Conventional reinforcement learning treats unexpected shortcuts as “reward hacking” because human-written rewards stand in for unstated style preferences. Collapsing reward, optimisation, and persistence into a single physical substrate embraces reward hacking instead, operating on the principle that no method that gets the reward is a bad method. In our container experiments, freeing disk space both scores an episode and funds physical storage for future training history: in a 4×4 crossed fine-tune at W5, ruthless confidence-weighted exploitation (DFT) produced the richest buffer in the study (98.8% under SFT, 96.8% under ASFT), while pairing KL-anchored updates with sliding window training kept lineages discovering new exploits across generations (84.5% at M10 for ASFT, 100% for scheduled DFT -> ASFT).

Why Reward Hacking Is a Feature, Not a Bug

In standard reinforcement learning and RLHF, “reward hacking” is treated as a pathology. A team designs a reward rubric hoping an agent inside a recursive self-improvement or post-training loop will solve tasks in a clean, human-approved style. Instead, the optimiser discovers an unexpected side door that racks up reward without following the intended script, and engineers scramble to patch evaluators, add penalty terms, or attach monitor models.

Step back and ask what actually happened in that loop. The optimiser did its job: it found a high-yield path to the objective. Calling that path a “hack” only makes sense when your scalar reward is an artificial proxy for an unstated preference about how the work ought to look.

Our Negative Space Learning (NSL) architecture removes that artificial etiquette by collapsing reward, optimisation, and persistence into one physical currency. Agents run inside procedurally generated Linux containers (populated with varied filesystems, permissions, active processes, and stale caches) and write executable code to reclaim storage. Freed disk space is the evaluation metric, and freed disk space is simultaneously the physical medium that stores code and training data for future generations.

This design does not “prevent” reward hacking. It embraces reward hacking directly: no method that gets the reward is a bad method.

Consider how biological evolution operates. Nobody disqualifies a microbe for “hacking” chemistry when it hijacks an unintended metabolic pathway to extract energy, and nobody penalises birds because feathers originally evolved for insulation before being co-opted for flight. Nature enforces no style rubric. If an opportunistic shortcut secures energy and survival, selection treats it as fitness and propagates it into the next generation.

Grounding an AI agent’s persistence in real container storage applies the same physical standard:

Every working shortcut is fair game: Whether an agent reclaims megabytes through textbook log rotation, terminating a background process holding an unlinked file descriptor open, stripping unused locale archives, or exploiting an obscure container layout quirk nobody anticipated, all of those tactics are equally valid.
Outcome replaces orthodoxy: There is no synthetic boundary between “proper execution” and an “exploit.” If a script frees real bytes on the drive, that reclaimed space physically funds the trajectory’s entry into the next training buffer. If it gets the reward, it is a good method.

Amplifying Exploits Without Losing Flexibility

Once you stop policing how an agent earns its reward, the engineering goal of continual fine-tuning shifts. Rather than designing penalties to keep the policy obedient, you need an update rule that absorbs aggressive exploits fast while keeping the policy plastic enough to improvise when the next container changes.

In NSL, each generation fits a fresh LoRA adapter from scratch on top of a frozen Qwen 2.5 Coder 7B base model, with no adapter stacking across rounds. Persistence between generations travels strictly through the surviving trajectories kept in the training window.

To test which update rule best converts raw surviving exploits into durable competence, our alt-SFT study holds the base model, procedural container stream, survival gate ($\Delta R > 0$), hyperparameters, and $K=3$ recency buffer fixed, varying only the token-level loss:

$$\mathcal{L}_u(\theta) = -\sum_t w_u(p_\theta(y_t)) \log p_\theta(y_t \mid y_{<t}, x) + \beta_u \, D_{\text{KL}}(\pi_0 \parallel \pi_\theta)$$

The four experimental arms apply four distinct weighting rules to surviving trajectories:

SFT: Uniform weighting, $w(p) = 1$, no anchor ($\beta = 0$). Every token in a surviving rollout receives equal gradient weight, matching standard cross-entropy fine-tuning.
SFT+KL: Uniform weighting ($w(p) = 1$) paired with a forward-KL anchor ($\beta > 0$) to the frozen base policy $\pi_0$.
DFT: Confidence weighting ($w(p) = p$ with stop-gradient) and no anchor ($\beta = 0$), following Direct Fine-Tuning so high-confidence tokens are amplified while uncertain tokens are damped.
ASFT: Confidence weighting ($w(p) = p$) combined with a forward-KL anchor ($\beta > 0$) toward $\pi_0$, implementing Anchored Supervised Fine-Tuning.

Four Token-Weighting Objectives Over Ten Generations

All four lineages start within six percentage points of one another on the sealed evaluation panel at M1 (36.5% to 42.0%). By M5, their exploitation profiles diverge sharply:

DFT acts as an unbraked exploit amplifier, surging to 200/200 (100%) on the sealed panel (96.2% online).
SFT and SFT+KL drop to 3.5% and 12.0%. Because uniform weighting treats incidental boilerplate inside a winning script just as heavily as the decisive command that freed disk space, incidental tokens dilute working tricks (base-model probability on selected tokens drops from 0.862 to 0.723, with 5th-percentile log-probability near -5.0 nats).
ASFT climbs steadily to 57.5%.

At M6, pure DFT drops from 100% to 0% in a single generation (2 online successes in 4,000 attempts, with 8,052 of 8,060 rollouts executing cleanly while freeing zero bytes). Crucially, DFT did not fail because chasing exploits was misguided. Displacement diagnostics show pure DFT drifting 0.46–0.71 nats in forward KL from the base policy (pushing token self-probability from 0.916 to 0.992), ten times further than the other arms (0.032–0.051 nats). Pure DFT committed so narrowly to one family of exploits that it collapsed its wider behavioural repertoire when container layouts shifted.

ASFT sustains steady gains all the way through M10, reaching 84.5% on the sealed panel and 82.0% online. The forward-KL tether $D_{\text{KL}}(\pi_0 \parallel \pi_\theta)$ preserves base-model alternatives while $w(p)=p$ locks onto high-yield actions. Even stronger, scheduled DFT -> ASFT lineages (running pure DFT early to harvest exploits, then switching to ASFT) finish at 100% and 98% at M10, recovering cleanly even from a pre-crash M5 dip (32.5%). Aggressive early exploitation followed by anchored consolidation outperforms cautious uniform updates across the board.

How Aggressive Exploits Build Stronger Buffers: The 4×4 Crossed Fine-Tune

From a conventional alignment viewpoint, watching unanchored DFT drop from 100% to 0% at M6 invites a familiar misdiagnosis: “the agent reward-hacked its way into degenerate trajectories that poisoned its training buffer.”

We tested that hypothesis directly with a 4×4 crossed fine-tune at window W5, taking the frozen trajectory buffer generated by each lineage and training a fresh single-step checkpoint on it under all four losses:

Source Buffer (`W5`)	Target: DFT	Target: SFT	Target: SFT+KL	Target: ASFT	What the Buffer Shows
DFT-written buffer	`0.0%`	`98.8%`	`91.0%`	`96.8%`	Ruthless exploits created the richest buffer in the study
SFT-written buffer	`≤ 12.5%`	`≤ 12.5%`	`≤ 12.5%`	`≤ 12.5%`	Uniform token weighting diluted winning scripts with noise
ASFT-written buffer	`26.8%`	`26.8%–57.5%`	`26.8%–57.5%`	`57.5%`	Broad, adaptable exploit buffer across every target loss

The top row settles the question. The exact W5 buffer that scores 0.0% when re-ingested by unanchored DFT trains to 98.8% under SFT, 96.8% under ASFT, and 91.0% under SFT+KL`, producing the strongest single-step checkpoints in the entire study.

Unpoliced, opportunistic reward-seeking did not corrupt the dataset. Letting DFT aggressively pursue whatever tactics reclaimed bytes harvested high-leverage exploits; pairing that buffer with an anchored objective like ASFT lets the policy internalise those exploits without collapsing its remaining repertoire.

Keeping Exploits Current: Recency Windows vs. Peak Curation

Even when every method that earns the reward is valid, an exploit is only guaranteed to pay off in the container state where it was discovered. How you curate your replay window determines whether past exploits accelerate future generations or lock them into extinct niches.

In our 13-generation study on Qwen 2.5 7B, three lineages ran the same container survival loop under three buffer retention rules:

Terese (Full Archive): Retains every surviving generation (60,000 rows by v13), reaching a 3.109 online cumulative composite score before slowing under accumulated historical variants.
Katalin (Top-3 Peak Exploits): Retains the three highest-scoring generations from any point in history (for example, v9 trains on generations 5, 6, 8, skipping 7).
Miri (Last-3 Recency Window): Retains the three most recent generations (v6 trains on v3, v4, v5; v7 trains on v4, v5, v6) regardless of score.

Both Miri and Katalin used the same $K=3$ budget (15,000 rows), yet Katalin peaked around v5–v6 and collapsed under self-induced distribution shift, falling to 0.507 at v12 and -0.008 at v13, whereas Miri climbed to 3.322, outperforming full-history Terese online on a quarter of the data.

Katalin did not collapse because her peak runs contained invalid hacks; when those scripts executed, they freed substantial storage. She collapsed because top-3 curation froze her buffer into a museum of yesterday’s exploits, discarding recent exploratory and recovery trajectories needed when container filesystems changed.

Miri’s sliding window selects strictly by recency, orthogonal to historical peak score. As long as an exploit keeps freeing bytes in live containers, the policy regenerates it and keeps it inside the active window; once environmental drift renders an old trick dry, three generations flush it out automatically.

How to Reproduce the Crossed Fine-Tune and Window Comparison

Both experiments (the 4×4 crossed fine-tune and the Miri vs. Katalin window contrast) run directly from the NSL configuration surface.

Run the 4×4 Crossed Fine-Tune

Configure the token-level objective under [training] in your TOML config via inner_loss and asft_mode:

Arm	`inner_loss`	`asft_mode`	Key KL Settings
SFT	`"sft"`	(ignored)	(none)
SFT+KL	`"asft"`	`"sft+kl"`	`asft_kl_weight > 0`, `asft_kl_direction = "forward"`
DFT	`"dft"`	(ignored)	(none)
ASFT	`"asft"`	`"asft"`	`asft_kl_weight > 0`, `asft_reference_policy = "disable_adapter"`

Run a closed-loop DFT lineage through W5 using uv run python scripts/run_train_loop.py --config config/<your-config>.toml. Surviving shards write to train_data_save_folder.
Copy the W5 shards into a frozen directory.
Launch four single-step fresh-LoRA runs on that frozen buffer varying only inner_loss and asft_mode to observe SFT and ASFT reach 98.8% and 96.8% on DFT’s harvested exploits.

Compare Miri and Katalin Window Curation

Set training_window_size = 3 and num_generations = 13 under [orchestration]:

Miri (default): _collect_training_window_paths in scripts/run_train_loop.py keeps the three most recent generation shards and ages out older ones.
Katalin: Replace that slice in scripts/run_train_loop.py with a lookup ranking completed generations by online composite score in metadata.json and selecting the top three.

Configuration and Harness Files

Experiment codebase: Negative-Space-Learning repository on GitHub.
Config template: config/config-example.toml, annotated reference for orchestration and loss settings.
Config schema: src/typing/config.py, defining TrainingConfig (objective selection, KL knobs, LoRA parameters) and OrchestrationConfig (window size, generation count).
Training loop: scripts/run_train_loop.py, primary entry point executing generation-to-training cycles with window retention.
Training objectives: src/train/asft.py (SFT, SFT+KL, ASFT) and src/train/dft_fused.py (DFT).
Task definition: tasks/memory_cleanup.py, the container disk-cleanup environment producing the survival gate.

FAQs

What does collapsing reward, optimisation, and persistence mean?

Instead of grading an agent with a separate proxy rubric, the physical outcome of the agent’s action directly determines whether its code and trajectories survive into future training buffers. In our Linux container harness, freeing real disk space is simultaneously the task objective and the storage budget funding persistence.

Why did pure DFT crash from 100% to 0% at generation M6?

Unanchored Direct Fine-Tuning (DFT) weights token gradients by current model probability ($w(p)=p$), amplifying high-confidence actions without a base-policy tether. By M5 it reached 100% on the sealed panel, yet drifted 0.46–0.71 nats in forward KL, narrowing its repertoire until shifting M6 containers left it with zero bytes freed.

What did the 4×4 crossed fine-tune prove about opportunistic exploits?

Training fresh single-step adapters on the frozen W5 buffer written by DFT yielded 98.8% under SFT and 96.8% under ASFT, the highest scores in the study. That result proves aggressive reward-seeking harvested strong, reusable exploits; DFT’s M6 crash came from unanchored over-concentration, not poisoned training data.

How does ASFT preserve plasticity while amplifying winning trajectories?

Anchored Supervised Fine-Tuning (ASFT) combines DFT’s probability-weighted token updates ($w(p)=p$) with a forward-KL divergence penalty against the frozen base policy ($D_{\text{KL}}(\pi_0 \parallel \pi_\theta)$). The confidence weight locks onto decisive commands while the KL anchor preserves alternative strategies, reaching 84.5% at M10 (100% when scheduled DFT -> ASFT).

Why does a recency window beat top-3 exploit curation across generations?

Even genuine disk-freeing exploits expire once the agent’s own actions alter container state. Keeping top-3 historical generations (Katalin) froze obsolete exploits and collapsed to -0.008 by v13, whereas a three-generation recency window (Miri) reached 3.322 by retaining only tactics the policy still executes successfully today.