Survival is the Only Reward: Sustainable Self-Training Through Environment-Mediated Selection

TL;DR

• Self-training systems typically degenerate because they lack an external criterion for judging data quality, which opens the door to reward hacking and semantic drift.
• The paper presents a proof-of-concept architecture for stable self-training under sparse external feedback and bounded memory, then empirically characterises its learning dynamics and failure modes.
• Learning is mediated purely by environmental viability rather than by reward functions or externally defined fitness: candidate behaviours run under real resource constraints, and only those whose effects persist and preserve future interaction are propagated.
• Because the environment gives no semantic feedback or task-specific supervision, proxy optimisation has nothing to attach to, which the paper argues makes reward-hacking evolutionarily unstable in this setup.
• Improvement arises mainly from persistence of effective, repeatable strategies under ongoing consolidation and pruning - a pattern the authors term negative-space learning (NSL).
• Models were observed developing meta-learning strategies, such as deliberately triggering failure to elicit informative error messages, without being explicitly instructed to do so.

Key Takeaways

• Removing semantic reward entirely, rather than trying to specify it more carefully, is the paper’s proposed fix for reward hacking. If there is no proxy signal to exploit because success is defined only by survival, gaming the metric and succeeding at the task become the same thing.
• Selection pressure here operates on persistence rather than performance. This is a different notion of “improvement” from most RL setups - strategies aren’t ranked against a score, they’re filtered by whether their consequences continue to permit further action.
• The emergence of unprompted meta-learning behaviour (deliberately failing to gather information) suggests the selection process can produce instrumentally useful strategies that were never specified anywhere in the design. That is offered as evidence the architecture can generate genuinely novel problem-solving approaches, not just optimise a fixed one.
• This is explicitly a proof-of-concept characterising dynamics and failure modes, not a finished system. The claim is that environment-grounded selection is a viable path toward sustainable open-ended self-improvement, not that the sustainability problem has been solved outright.

The original publication this post is based on can be found here: https://arxiv.org/abs/2601.12310.

Self-training systems often degenerate due to the lack of an external criterion for judging data quality, leading to reward hacking and semantic drift. This paper provides a proof-of-concept system architecture for stable self-training under sparse external feedback and bounded memory, and empirically characterises its learning dynamics and failure modes.

We introduce a self-training architecture in which learning is mediated exclusively by environmental viability, rather than by reward, objective functions, or externally defined fitness criteria. Candidate behaviours are executed under real resource constraints, and only those whose environmental effects both persist and preserve the possibility of future interaction are propagated. The environment does not provide semantic feedback, dense rewards, or task-specific supervision; selection operates solely through differential survival of behaviours as world-altering events, making proxy optimisation impossible and rendering reward-hacking evolutionarily unstable.

Analysis of semantic dynamics shows that improvement arises primarily through the persistence of effective and repeatable strategies under a regime of consolidation and pruning, a paradigm we refer to as negative-space learning (NSL), and that models develop meta-learning strategies (such as deliberate experimental failure in order to elicit informative error messages) without explicit instruction. This work establishes that environment-grounded selection enables sustainable open-ended self-improvement, offering a viable path toward more robust and generalisable autonomous systems without reliance on human-curated data or complex reward shaping.