Survival is the Only Reward: Sustainable Self-Training Through Environment-Mediated Selection

TL;DR

• Self-evolving agents usually degrade because nothing outside the model judges their data, which invites reward hacking and semantic drift. This paper removes the reward function and selects behaviours only by whether they survive in a real environment.
• Three Qwen 2.5 7B Instruct lineages trained on their own surviving code for 13 versions. The one limited to its three most recent datasets kept improving and reached over 50% of the performance of a lineage trained on four times as much data.
• Most gains came from pruning what did not work, which we call negative-space learning (NSL), and the models learned to fail on purpose to get informative error messages without being told to.

Key Takeaways

• Removing semantic reward entirely, rather than trying to specify it more carefully, is the paper’s proposed fix for reward hacking. If success is defined only by survival, there is no proxy signal to exploit: gaming the metric and succeeding at the task become the same thing.
• Selection pressure here operates on persistence rather than performance. Strategies are not ranked against a score. They are filtered by whether their consequences still permit further action.
• Unprompted meta-learning suggests the selection process can produce useful strategies nobody specified. We offer it as evidence that the architecture can find new problem-solving approaches, not just optimise a fixed one.

This post summarises our paper on training self-evolving agents through environment-mediated selection, published on arXiv at https://arxiv.org/abs/2601.12310.

Paper abstract

Self-training systems often degenerate due to the lack of an external criterion for judging data quality, leading to reward hacking and semantic drift. This paper provides a proof-of-concept system architecture for stable self-training under sparse external feedback and bounded memory, and empirically characterises its learning dynamics and failure modes.

We introduce a self-training architecture in which learning is mediated exclusively by environmental viability, rather than by reward, objective functions, or externally defined fitness criteria. Candidate behaviours are executed under real resource constraints, and only those whose environmental effects both persist and preserve the possibility of future interaction are propagated. The environment does not provide semantic feedback, dense rewards, or task-specific supervision; selection operates solely through differential survival of behaviours as world-altering events, making proxy optimisation impossible and rendering reward-hacking evolutionarily unstable.

Analysis of semantic dynamics shows that improvement arises primarily through the persistence of effective and repeatable strategies under a regime of consolidation and pruning, a paradigm we refer to as negative-space learning (NSL), and that models develop meta-learning strategies (such as deliberate experimental failure in order to elicit informative error messages) without explicit instruction. This work establishes that environment-grounded selection enables sustainable open-ended self-improvement, offering a viable path toward more robust and generalisable autonomous systems without reliance on human-curated data or complex reward shaping.

How to train self-evolving agents without a reward function

A model that trains on its own output needs something outside itself to decide which outputs to keep. Our earlier paper, Generalising from Self-Produced Data, proposed an ungameable real-world number for that job; this one tests the idea over 13 model versions. The paper sets three requirements for that filter. It must be:

1. External to the model’s internal representations.
2. Causally downstream of the model’s actions.
3. Hard, or counterproductive, for the model to manipulate.

None of these asks the filter to know what a correct answer looks like. It only has to let behaviours survive or disappear according to their real consequences.

Our filter is non-volatile storage space. Space is what an agent needs to keep copies of itself and its training data, so it is survival in a literal sense, not a score that stands for survival. Reward, optimisation and persistence collapse into one quantity. A strategy that games the measurement without actually gaining space uses up the resource its future depends on, so it dies out on its own.

The environment and training loop

The agent works inside procedurally generated networks of Linux containers, with realistic files, folders, permissions and passwords. It starts with access to one container and has to work out for itself how to reach the others. When it has taken all the available space, or broken the system beyond use, a new network is generated. Each network differs in detail but keeps the same underlying structure, so behaviour can be compared across iterations.

The loop itself is simple:

1. The model proposes a behaviour as executable code.
2. The code runs in the environment.
3. The environment changes in a measurable way: storage is freed or lost.
4. The behaviour and its outcome are recorded.
5. Only behaviours with a net gain in usable space are kept for training.

Each iteration produced 4,500 rows of data, topped up with 500 rows of rehearsal data from the Stack Exchange Paired dataset. We tried PPO, GRPO and DPO, but plain supervised fine-tuning on successful trajectories was the most stable. To keep training indefinitely without catastrophic forgetting, each new dataset trains a fresh LoRA applied directly to the base model, rather than stacking LoRAs on top of each other. The code is in our Negative-Space-Learning repository.

Three lineages of self-evolving agents

We ran the same pipeline on three identical Qwen 2.5 7B Instruct models for five iterations, then gave each lineage a different data regime up to version 13.

Terese: all data

Terese trained on every dataset produced so far, reaching 60,000 rows by version 13. It is the benchmark for how far pure data quantity goes.

Miri: three most recent datasets

Miri trained only on the data from the three iterations immediately before it. No version ever saw the full history, so an early discovery survived only if later versions kept using it.

Katalin: three best datasets

Katalin trained on the three earlier datasets with the highest share of code that ran successfully. It selected for peak performance, whenever that peak happened.

Miri is the result that matters most. It kept improving under a strict memory limit, and Miri v13 reached over 50% of the performance of Terese v13 with a quarter of the training data. Katalin went the other way: versions 12 and 13 kept degrading. Our reading is that choosing data by isolated peak scores kept individually strong strategies but eroded the basic skills needed to combine them in new situations.

Negative-space learning: improving by pruning

We set up the survival experiment to watch agents acquire new skills. What we saw instead was mostly subtraction. Comparing the strategies of version 1 and version 13 in each lineage, a few strategies were added, but far more were merged, refined, compressed or dropped.

We call this negative-space learning: the model improves by working out what does not work in its domain and shifting probability away from it. It is not the same as forgetting general skills. HumanEval scores stayed about the same or dropped only slightly while performance in the environment rose sharply. It also explains Miri. Keeping only the three most recent datasets means that only strategies good enough to be repeated across different environments survive, which works as a slow but thorough triage.

Meta-learning that nobody asked for

As performance improved, first-try success (pass@1) fell. By the later iterations, Terese and Katalin were scoring zero pass@1 during data generation, while producing a smaller share of code that failed to run.

The agents had found a use for failure. Running code that fails at the first attempt opens a debug cycle, and the debug cycle is extra room to explore. Nothing in the setup rewards exploration or information gain. The behaviour appeared because, in an environment with enough feedback, strategies that buy information for later turned out to survive. This is the kind of loop self-improving AI agents depend on: the system gets better at getting better, with no human judging each step.

Do the learned skills transfer?

Every lineage evolved in Linux containers. To check whether the gains were specific to that setting, we tested the base model and versions 2 and 13 of each lineage on a Windows-based Docker system none of them had seen.

The improvement is smaller than in Linux, but it is there in every lineage, which points to skills that generalise beyond the environment they were learned in.

FAQs

What are self-evolving agents?

Self-evolving agents are AI systems that improve by training on data they produce themselves, without a human curating that data. The hard part is deciding which outputs to keep. In this paper, the environment decides: an agent’s code runs in real containers, and only behaviours that gain storage space become training data for the next version.

What is negative-space learning?

Negative-space learning (NSL) is our name for improvement that comes mainly from pruning rather than adding. Across 13 versions, our agents added a few strategies but merged, compressed or dropped many more. The model gets better by learning what does not work in its environment and moving away from it, while keeping its general skills.

Why not use a reward function?

A reward function is a proxy, and a model can learn to satisfy the proxy without doing the task. Here the selection signal is storage space the agent needs to survive, so gaming the signal and doing the task are the same thing. A strategy that fakes a gain wastes the resource its own future depends on.