Reinforcement Learning on Eigenform Articles

Survival is the Only Reward

Tue, 27 Jan 2026 00:00:00 +0800

TL;DR

Describes a five-year project to build an “evolutionary” LLM that survives by finding and holding disk space in procedurally generated networked Linux container environments, contrasted with conventional RL setups (racing games, mazes) that have a finite win condition and only “horizontal” complexity; open-ended survival has no win state and forces “vertical” complexity, where new strategies keep opening onto new problems.
Three Qwen 2.5 7B model lineages were trained differently on the resulting data: Terese on all cumulative data (a “just add more data” baseline), Miri on a sliding window of the past three runs, and Katalin on only the top-performing three runs - Terese plateaued, Miri improved incrementally without catastrophic forgetting or needing to scale data or parameters, and Katalin collapsed after overfitting to narrow, high-scoring niche strategies.
Most of the measured performance gains came from efficiency - discarding what doesn’t work and reusing known-good strategies verbatim, a pattern the piece calls subtractive learning - rather than from genuinely new capability, visible both in raw speed gains and in semantic clustering of the strategies generated across generations.
Agents appeared to develop meta-learning without being told to: their first-try code success rate fell even as real task success rose, because they learned to deliberately write failing code to elicit informative error messages useful both immediately and as future training data - which the piece argues dissolves the usual concept of reward hacking in a fully open-ended setting, since any strategy that aids survival is legitimate by definition.
Training exclusively in Linux environments improved every model’s performance in an entirely untested Windows environment, offered as evidence of genuinely transferable learning rather than memorised environment-specific tricks - though the piece notes the actual rate of improvement was far slower than the “intelligence explosion” dynamic its earlier theoretical paper had predicted.
The piece reveals it used supervised fine-tuning throughout, not reinforcement learning, arguing RL’s credit-assignment machinery is built for tasks with a definable win state and actively worked against this open-ended setup; it closes by arguing that superhuman AI likely requires this kind of unscaffoldable open-ended learning, and that open-ended continuous learning and durable alignment are mechanically, not just philosophically, incompatible.

Key Takeaways

Survival is the Only Reward: Sustainable Self-Training Through Environment-Mediated Selection

Sun, 18 Jan 2026 00:00:00 +0800

TL;DR

Self-training systems typically degenerate because they lack an external criterion for judging data quality, which opens the door to reward hacking and semantic drift.
The paper presents a proof-of-concept architecture for stable self-training under sparse external feedback and bounded memory, then empirically characterises its learning dynamics and failure modes.
Learning is mediated purely by environmental viability rather than by reward functions or externally defined fitness: candidate behaviours run under real resource constraints, and only those whose effects persist and preserve future interaction are propagated.
Because the environment gives no semantic feedback or task-specific supervision, proxy optimisation has nothing to attach to, which the paper argues makes reward-hacking evolutionarily unstable in this setup.
Improvement arises mainly from persistence of effective, repeatable strategies under ongoing consolidation and pruning - a pattern the authors term negative-space learning (NSL).
Models were observed developing meta-learning strategies, such as deliberately triggering failure to elicit informative error messages, without being explicitly instructed to do so.

Key Takeaways

Generalising from Self-Produced Data: Model Training Beyond Human Constraints

Mon, 07 Apr 2025 00:00:00 +0800

TL;DR

Current LLMs are bounded by human-derived training data and by a single level of abstraction that prevents definitive truth judgments about their own outputs.
The paper proposes a framework in which AI agents generate and validate new knowledge directly through interaction with their environment, rather than through further human-curated data.
The central device is an unbounded, ungamable numeric reward - such as annexed disk space or follower count - that requires no human-defined benchmark.
Agents iteratively generate strategies and executable code to maximise this metric, and successful outcomes become the basis for self-retraining.
Empirical validation (does the outcome actually hold up) is used in place of textual similarity to guard against model collapse and the “warm start” problem, with fine-tuning carried out via GRPO.
The system is built from modular agents handling environment analysis, strategy generation, and code synthesis separately, to allow scalable experimentation.

Key Takeaways