How Self-Improving AI Agents Work: Feedback Loops, Memory, Skills and Evaluation

What makes an agent self-improving

The fundamental question in agent architecture is what carries over from one run to the next. In standard practice, three distinct levels exist:

• Nothing. A normal agent. You can raise its ceiling with better prompts and tools, but you have to raise it manually each time.
• Context. Retrieval and memory files let an agent carry facts across sessions. That is remembering, not improving. The model underneath remains unchanged.
• The policy. The agent updates its own weights or code from outcomes, so what it learns in one round is permanently there for the next. This is the version that compounds.

For the third kind to improve rather than drift, three conditions have to hold:

The three conditions for compounding improvement

1. Autonomous Modification. The agent must apply the change itself, without waiting on an external operator or human in the loop during each update cycle. A model that only improves when a human manually retrains it is not self-improving in the relevant sense, however good the resulting model is.

2. Method-Level Change. The change must act on how the agent operates, not merely on a single output. Learning a static fact does not count; learning a more effective method to discover and verify facts does.

3. An Ungameable Check. The agent needs an objective verification signal it can trust that a candidate change is truly an improvement before keeping it. Most systems fail here. Compute and training data are rarely the limiting factor; a check you cannot cheat is.

A history of agentic self-improvement

The full lineage runs across symbolic AI, neural networks, control theory and artificial life, and is laid out on our recursive self-improvement page. Below is the agent-focused cut: the historical systems where an entity acts, encounters real outcomes, and rewrites itself.

The roots (1959 to 2011)

Arthur Samuel's checkers program at IBM was the first working example. It improved by playing against itself and adjusting its board evaluation weights based on outcomes (Samuel, 1959). Self-play remains one of the strongest training foundations today. In 1965, I. J. Good named the endpoint: a machine capable of building an ultra-intelligent machine takes over the research process, and human researchers stop setting the pace (Good, 1965). Douglas Lenat's EURISKO later placed heuristics in charge of rewriting their own heuristics, winning a national fleet-design tournament two years in a row before organisers altered the rules to prohibit it (Lenat, 1983).

Jürgen Schmidhuber's lab formalised these ideas mathematically. The Gödel Machine represents the strict theoretic form: it rewrites any part of its software or proof search, but only after producing a formal proof that the change increases global expected reward (Schmidhuber, 2003). PowerPlay subsequently dropped the requirement for formal proof, instead searching for the simplest problem the system cannot yet solve, paired with an internal modification that solves it without breaking prior capabilities (Schmidhuber, 2011)—one of the first architectures to autonomously curate its own learning curriculum.

Self-play reinforcement learning (1992 to 2020)

Gerald Tesauro's TD-Gammon achieved master-level backgammon play purely from self-play reinforcement learning, without requiring human gameplay databases (Tesauro, 1995). The modern form culminated in expert iteration (Anthony, Tian and Barber, 2017): DeepMind's AlphaGo Zero and AlphaZero mastered chess, shogi and Go starting from random initialisation, relying solely on terminal game results as feedback (Silver et al., 2018). MuZero extended this by planning without even being provided the underlying rules of the environment (Schrittwieser et al., 2020). The feedback signal does not need to be rich; binary win/loss outcomes suffice if they are completely trustworthy.

Architecture and algorithm search (2017 to 2020)

A parallel research line focused on optimising training pipelines rather than task responses. Neural Architecture Search (NAS) used recurrent controllers to generate model topologies, evaluating candidates by training them to convergence (Zoph and Le, 2017). Population-Based Training ran ensembles of models concurrently, allowing underperforming instances to clone and mutate parameters from high-performing peers mid-run (Jaderberg et al., 2017). AutoML-Zero pushed this to the foundational level, evolving machine learning algorithms directly from elementary mathematical operations (Real et al., 2020).

Language models that train themselves (2022 to now)

Once language models became competent code and text generators, they could be deployed to construct parts of their own training pipelines. Modern research divides into four primary surfaces:

• Generating data: Self-Instruct had models construct their own instruction datasets (Wang et al., 2022); STaR generated rationales, filtered for those yielding correct solutions, and fine-tuned on the verified subset (Zelikman et al., 2022).
• Generating judgement: Constitutional AI substituted human preference labels with self-critiques against explicit rules (Bai et al., 2022); Self-Rewarding Language Models trained LLMs to act as their own reward models in iterative loops (Yuan et al., 2024).
• Generating tasks: SEAL trained models to produce fine-tuning objectives and update instructions evaluated against downstream model accuracy (Zweiger et al., 2025); Absolute Zero Reasoner synthesised and solved coding challenges with zero external human data (Zhao et al., 2025).
• Generating experience: Eigenform's Negative Space Learning deployed agents into persistent Linux container networks where actions generated real survival feedback: successful strategies were preserved, failed attempts discarded, and surviving traces used to fine-tune subsequent generations (Dodgson et al., 2026).

Agents that rewrite their own code (2023 to now)

The latest frontier embeds language models inside evolutionary loops governed by rigorous code execution verifiers. FunSearch generated programs to solve complex mathematical problems, scoring candidates with an interpreter rather than an LLM judge, discovering novel mathematical constructions (Romera-Paredes et al., 2024). AlphaEvolve applied evolutionary search to production software, achieving verified optimisations in data-centre scheduling and matrix multiplication (Novikov et al., 2025). The Darwin Gödel Machine pushed further by modifying its own inspection code and decision logic, adopting changes only when validated against benchmark test suites (Zhang et al., 2025).

The four things a self-improving agent needs

1. Feedback loops

Each cycle requires objective scoring based on unforgeable consequences: executable code that compiles and passes unit tests, a game that terminates in victory, or physical disk space freed. Proxy signals, such as LLM judges, can be gamed. In recursive compounding loops, a gameable proxy is worse than no feedback at all, because the agent systematically learns to exploit the discrepancy between the proxy and real utility.

2. Durable Memory

Improvements must persist across sessions. Ephemeral scratchpads expire when sessions terminate. Context retrieval retains episodic facts, not procedural skills. Only modifications to model weights or executable code carry capabilities forward into subsequent generations. Most commercially labelled "self-improving" assistants merely store conversational history, which is why they fail to exhibit compounding capability.

3. Generalising Skills

A narrow agent improves only within a single task distribution. A general agent improves its meta-learning capacity: acquiring one skill accelerates the acquisition of the next because novel challenges share structural overlap with previously solved domains.

4. Objective Evaluation

Evaluating a self-improving system is inherently challenging because the policy evolves dynamically. In our disk-space survival experiments, first-attempt code accuracy actually declined across generations even as overall task success rose: agents deliberately learned to write failing exploratory probes to extract informative system error diagnostics. Rigorous instrumentation is essential to distinguish tactical error-generation from model regression.

How our geology systems work

At Eigenform, we test self-improving agent architectures on mineral exploration datasets because natural geological environments are exceptionally demanding. Exploration data is fragmented, acquiring ground truth requires expensive physical drilling, and superficial correlation easily collapses upon real testing.

The mechanism: environment-mediated selection

Described in Survival is the Only Reward (arXiv:2601.12310), agents inhabit networked, disposable Linux container clusters and compete to secure disk space. Disk capacity represents an unforgeable physical metric. Successful container actions are captured into supervised fine-tuning corpora, updating model weights for the next generation. We observed that moderate forgetting (using sliding-window training) outperformed cumulative historical retraining by shedding obsolete local traps.

The geology application

Our geology system applies the same generate-then-train loop to subsurface geological exploration. To prevent reward gaming, the workflow is distributed across four isolated micro-agents with strict separation of duties:

The scoring metric is a Bayesian Information Criterion (BIC) evaluated over a three-dimensional voxel grid. Features are cross-validated across held-out depth slices, testing whether surface geochemistry genuinely predicts deep mineralization. Candidate features are retained only if they improve the compression of the entire voxel model, in accordance with the principles described in Building a New World Model via Coherence Mapping.

In compounding loops, a gameable proxy is worse than no feedback at all: the agent systematically learns to exploit the gap between the score and reality.

Who is building self-improving AI agents

Research in self-improving architectures remains concentrated among leading specialized laboratories: Google DeepMind (AlphaEvolve, FunSearch, AlphaZero), Sakana AI (Darwin Gödel Machine), Schmidhuber's lab at IDSIA (Gödel Machine, PowerPlay), and elite academic groups exploring test-time adaptation. Most current efforts focus on competitive programming and formal mathematics where ground truth can be checked with compiler tooling. For a complete audit of what has been empirically demonstrated across outputs, harnesses, and weights, see our analysis in Self-Improving AI: What Has Actually Been Demonstrated.

Eigenform focuses specifically on eliminating the dependency on human-crafted synthetic rubrics. By grounding agents in complex real-world environments—such as exploration geoscience archives and Linux operating systems—we demonstrate how agents learn to improve when the physical environment itself serves as the ungameable arbiter of truth.

Frequently Asked Questions