TL;DR
A self-improving agent architecture is the arrangement of parts that lets an agent change itself and know whether the change was an improvement. It is useful to read any such system as four components: something that acts, something that records, something that judges, and something that can undo. The last of these is the one most treatments leave implicit. The harder design problems are in the judging component, because the loop cannot be better than the signal supervising it.
Key Takeaways
- Read the architecture as four components: an actor, a trace or memory store, an evaluator, and a reversion mechanism. Most frameworks name the first three; treating reversion as a peer is the argument here.
- The evaluator sets the ceiling. Whatever judges the agent’s output decides how much the agent can improve.
- Verification has a hierarchy: formal verifiers are strongest, then executable tests, then learned judges, then the agent’s own assessment. Loops built on the weakest rung confirm themselves.
- Memory is a mutable surface. Retrieval and episodic memory change behaviour with no weight update, which makes them easy to miss when auditing drift.
- What the agent may modify is a design choice about blast radius. Prompts, tool selection, memory contents, harness code and policy weights carry five very different risks.
- Reversion is not optional. Without state snapshots and a rollback path, an architecture cannot separate improvement from drift after the fact.
- At least one signal must sit outside the loop, measured in a pipeline the agent cannot influence.
- No current architecture closes the outermost loop. Research direction-setting remains the rung where humans stay.
What is a self-improving AI agent architecture?
A self-improving AI agent architecture is the arrangement of components that lets an agent change some part of how it works, using what it encounters while running, and check whether the change was an improvement before keeping it. A useful decomposition has four parts: an actor, a record, a judge, and an undo. The first three match what most agent frameworks already name; reversion is the one this post treats as first-class. This post is a supporting piece under our page on self-improving agents; read that first for the wider picture.
What matters here is loop closure rather than autonomy. A scheduled retraining job is human-mediated; a closed-loop AI agent applies the update to the next task itself, which means the architecture has to carry the machinery to decide whether that update was good.
The four agent architecture components
The decomposition this post uses has four parts wired into a feedback-loop architecture: an actor, a trace and memory layer, an evaluator, and a reversion layer. Here is what each part does, and how it fails when it is weak or missing.
The actor
The actor produces the work: a model, a prompt, a set of tools, and the scaffold that routes between them. Most design attention lands here because it is the visible part. It is also the part where an architecture is least likely to fail, because a weak actor produces obviously weak output. What fails quietly is everything downstream of it.
The trace and memory layer
The trace records what the agent did and why: the inputs it saw, the intermediate steps, the outcome. Memory is the subset of that trace the agent carries into later tasks, whether as retrieval, an episodic store, or a summarised context. Agent memory architecture looks like a storage problem and is usually built as one. In a self-improving loop it works more like a control surface: what the agent retains shapes what it does next.
Memory is a mutable surface. An agent that stops surfacing certain examples has changed its effective knowledge base without a single weight moving. When auditing an agent for drift, the trace is where the evidence is, and a thin or unmanaged trace makes drift impossible to attribute after the fact. Our note on how not to build a slop cannon argues the same point from the other direction: chat history should be an explicit variable the harness manages, not something that passively accumulates.
The evaluator
The evaluator decides whether a candidate change is an improvement. It is the component that turns “the agent did something different” into “keep it” or “roll it back”. Evaluator and verifier design carries most of the architecture’s real decisions: how far the loop can improve is fixed by what the evaluator can catch.
The reversion layer
The reversion layer is the ability to put the agent back the way it was: state snapshots, versioned prompts and scaffolds, weight checkpoints, and a path to restore any of them. Rollback and checkpointing are what keep an agent auditable: without them, an architecture cannot separate improvement from drift once a change is in place, because there is nothing to compare against and no way back.
Where this sits in agent architecture
The classical picture of agent architecture is a taxonomy of control styles: reactive agents that map perception to action, deliberative agents that plan over a world model, and the hybrid and layered designs in between. Recent LLM-agent work adds a catalogue of orchestration patterns — a single agent with tools, a planner that decomposes, a critic that reviews, several agents handing work between them.
The self-improvement loop is orthogonal to all of these. Whatever control style or orchestration pattern the actor uses, the trace, the evaluator and the reversion layer are what turn a fixed agent into one that changes with use. A 2026 survey of agent systems (arXiv:2601.01743) organises the field around a similar set of parts: a policy core, a memory, and a critic that validates before acting. It does not treat rollback as a component; this post does. The emphasis here is on the evaluator and the out-of-band signal, since that is the part most likely to be wrong.
What the agent is allowed to change
What an agent may modify is a design decision, made surface by surface. The five surfaces behave very differently:
| Surface | Reversibility | Attribution difficulty | Blast radius |
|---|---|---|---|
| Prompt text | Trivial: version the string | Easy: one change, one effect | Narrow: one task type |
| Tool selection | Trivial | Easy | Narrow to moderate |
| Memory and retrieval contents | Moderate: needs snapshots | Hard: the effect is diffuse | Moderate: shifts many outputs at once |
| Harness and scaffold code | Moderate: version control | Moderate | Wide: every run |
| Policy weights | Hard: needs checkpoints and compute | Very hard | Wide and, without a checkpoint, permanent |
An architecture that lets the agent edit its own weights but keeps no checkpoints has chosen the widest blast radius with the weakest reversion. That can be a legitimate choice. It should be a deliberate one.
The evaluator problem
Verification has a hierarchy. Formal verifiers, where a proof settles whether a change is correct, are the strongest and the rarest. Executable tests, where code compiles and passes or it does not, are close behind and cover most real work. Learned judge models are weaker: they drift, they can be gamed, and they carry the biases of whatever trained them. Weakest is the agent assessing its own output, which produces a loop that confirms whatever the agent already believes.
Loops supervised only by a learned or intrinsic judge degrade in characteristic ways. Diversity collapses as the agent converges on whatever the judge rewards. Errors compound, because a gameable proxy is worse than no feedback: the agent learns to exploit the gap between the proxy and real utility, and does so faster with every cycle.
The structural fix is an out-of-band signal: at least one measurement the agent cannot influence, taken in a pipeline it does not control. In the exploration-analytics harness described in that piece, every place name and drillhole arrives already tagged with its true distance from the licence boundary, computed from coordinates before the model sees a word, and a separate pass re-reads the finished report against the same ground truth. The model writes; deciding what is inside the fence is not left to it. That is what an out-of-band evaluator looks like in practice.
Architectures in practice
Read each of these as an architecture: what acts, what judges, what can be undone.
- AlphaEvolve (Google DeepMind, 2025). The actor is a model proposing program edits inside an evolutionary loop. The evaluator is an automatic, executable scorer, near the top of the hierarchy. Reversion is implicit in the program database: a low-scoring candidate stays on record but is not selected as a parent.
- The Darwin Gödel Machine (Zhang et al., 2025). The actor edits its own codebase, the tools and workflow it uses to solve tasks. Every variant is scored on a coding-benchmark suite before it enters the archive. That archive of prior agents is also the reversion mechanism: the search can branch from any earlier agent, not only the current best.
- DSPy (Khattab et al., 2023). The actor is a declarative pipeline of model calls. The evaluator is a metric the developer supplies over a training set, and the system compiles prompts and few-shot examples against it. What is under optimisation is usually the scaffold rather than the weights.
- STaR (Zelikman et al., 2022). The actor generates reasoning chains. The evaluator is whether the final answer matches the known-correct one. The trace of successful chains becomes the next training set, so here the improvement lands on the weights.
For what these systems have and have not demonstrated, see our audit of self-improving AI.
What an AI agent self-improvement architecture cannot give you
A sound architecture buys you a loop you can trust to keep or discard its own changes. The limits are in what surrounds the loop.
It cannot manufacture grounding: a loop trained round after round on its own output, with no external contact, narrows toward a degenerate distribution however well its components are wired.
The compute cost does not go away either. Evaluating enough candidates for a search to be worthwhile is expensive, which is the reason this work relies on automatic evaluators.
And it cannot choose the problem. Every architecture here is handed its task and its verifier by a person; deciding what to work on next is the outermost loop, and the one humans still hold. That question is the subject of recursive self-improvement, unproven for the same reason.
Frequently Asked Questions
What is a self-improving agent architecture?
The arrangement of parts that lets an agent change some aspect of how it works and check whether the change helped: an actor that produces output, a trace or memory store, an evaluator that scores results, and a reversion path. The names differ between frameworks; the four roles stay the same.
What are the components of a self-improving agent architecture?
Four. An actor produces the work. A trace or memory layer records what happened and what was carried forward. An evaluator judges whether a change was an improvement. A reversion mechanism rolls back the changes that were not. Everything else is implementation detail.
What is the difference between a self-improvement and a self-learning agent architecture?
Self-learning usually means training a model on unlabelled data, which is a way of building the model. A self-improving architecture is a loop the deployed agent runs on itself: it changes something about how it works and verifies the result against a signal before keeping it.
What is an evaluator in agent architecture?
The component that decides whether a candidate change is an improvement. It can be a formal verifier, an executable test suite, a learned judge model, or the agent’s own assessment. The strength of the evaluator sets the ceiling on how far the loop can improve.
Can a self-improving agent modify its own code?
Some can. The Darwin Gödel Machine edits its own decision logic and keeps changes that pass a benchmark. Whether an agent may modify its harness, its prompts, its memory or its weights is a design choice about blast radius, not a question of capability.