Key Takeaways

• The Exogenous Shift Fallacy: Standard continual learning assumes the environment shifts externally while task distributions remain passive. In open-ended agent interaction, successful actions permanently mutate the environment, making the distribution shift endogenous and self-induced.
• Catastrophic Remembering: Accumulating all historical trajectories in a cumulative replay buffer (Terese) causes policy stagnation not from forgetting, but from averaging over mutually incompatible past states from destroyed worlds.
• The Sliding Window Solution: Retaining only the last three generations (\(K=3\)) of environment-validated trajectories (Miri) sustains monotonic improvement on a quarter of the data, achieving over 50% of peak performance without plateauing or collapsing.
• Epistemic Hygiene via the Environment: A sliding window does not destroy knowledge; it enforces epistemic hygiene. Valid strategies are naturally re-derived and re-enter the window through live environmental feedback, while obsolete habits age out cleanly.

TL;DR

When an AI agent’s actions permanently alter its environment, standard continual learning assumptions collapse. Cumulative replay buffers hoard trajectories from worlds the agent itself destroyed, causing catastrophic averaging over contradictory states. Testing identical Qwen2.5-7B-Instruct agents in procedural Linux containers reveals that accumulating all history (Terese) plateaus at 65,000 rows, while a \(K=3\) sliding window SFT buffer (Miri) discards ghost states and achieves continuous, monotonic improvement on 4x less data by letting the live environment re-derive valid strategies.

The Search Question

“How do you do continual learning when the agent’s own actions change the environment?”

Most of the field operates on a tacit assumption: the distribution shift is exogenous. The world changes, and the agent scrambles to keep up. Benchmarks like Split CIFAR or Rotated MNIST simulate this by handing the agent new, disjoint tasks. The agent is a passive observer of a shifting reality.

But in open-ended interaction, where an agent acts, explores, and survives, the shift is often endogenous. The agent clears the cache, and suddenly “clearing the cache” is no longer a valid move. The agent exploits a vulnerability, and the vulnerability closes. The agent changes the state of the world, and the world changes the distribution of future states.

We don’t have good benchmarks for this. We don’t even have good terminology for it.

If you ask, “Why do replay buffers fail in agent training?” you assume the answer (that they do fail) and you limit the scope to a specific mechanism. “Self-induced distribution shift in continual learning,” feels more like a philosophical question than a technical one (for now). 

But if you ask how to learn when your own actions break the ground you stand on, you stop treating the environment as a static dataset and start treating it as a dynamic system that you are actively disrupting. When a learning agent becomes good enough, at some point the major source of future world states becomes a function of that agent’s own world model.

Assumptions & Nuance

Standard continual learning makes three assumptions so ingrained they’re almost invisible. Each one breaks when the agent is inside the loop.

Assumption 2: Non-stationarity is exogenous. The world changes; the learner adapts. This is the premise behind every benchmark in the field. Split CIFAR hands you new classes. Rotated MNIST rotates the digits. The distribution shift arrives from outside the system, and the agent’s job is to track it.

But in open-ended interaction, the agent induces the shift. It clears the cache, and now “clear the cache” is no longer a valid move. It deletes the log files, and the log files stay deleted. The distribution doesn’t shift independently of learning: it shifts because of learning. The agent’s own successful actions permanently mutate the state it will encounter next.

Call this self-induced non-stationarity. It is the central reframe. The environment isn’t a river the agent stands beside; it’s a river the agent is damming, diverting, and drinking from. Every action rewrites the transition dynamics the agent will face on its next step.

Assumption 1: Past data is universal truth. The foundational CL instinct, encoded in replay buffers, elastic weight consolidation, and experience replay, is to prevent catastrophic forgetting by preserving what was learned. The past is treated as a repository of valid knowledge. Forgetting is the enemy.

In self-induced non-stationarity, this instinct is actively harmful. Past data doesn’t describe timeless regularities; it describes worlds the agent itself destroyed. The trajectory where clearing /var/log freed 200MB is still in the buffer, but /var/log is now empty. Training on that trajectory teaches the agent to expect a reward for an action that will produce nothing. A cumulative replay buffer doesn’t preserve knowledge: it averages over incompatible realities, producing a policy that is muddy and useless in all of them.

Assumption 4: Task boundaries exist. Even “task-free” continual learning assumes stationary interludes, periods where the distribution holds still long enough for the agent to consolidate. The shift is a discrete event: task 1 ends, task 2 begins.

When the agent alters the environment continuously, there are no interludes. The shift isn’t a discrete event tied to an external schedule; it’s tied to the agent’s own action history. Every successful action changes what comes next. The environment drifts on every timestep, and the drift rate is a function of the agent’s competence. The better the agent gets at freeing disk space, the faster the disk runs out of space to free.

What We Tested & Why

We were interested in agents that would priorise adaptability over perfection within an evolving environment. To investigate this in our self-training study (Survival is the Only Reward), we created an environment made up of procedurally generated network of Linux containers with realistic filesystems, real permissions, real processes, and real failure modes. The agent’s job is to free disk space. It explores the container, identifies strategies (clearing caches, deleting log files, compressing archives, killing orphaned processes) and executes them. Each action either frees space or it doesn’t. Only trajectories associated with a net-positive change in available storage are retained for training.

This is the loop: the agent acts, the environment records what happened, successful trajectories go into the training set, the model is fine-tuned, and the improved agent faces the environment again. Each generation, the cycle repeats.

The critical property of this setup is that successful actions permanently alter the state of the disk. When the agent deletes /var/log/syslog and frees 200MB, that file stays deleted. The next generation encounters a container where /var/log/syslog is already gone. The agent isn’t playing the same level again; it’s playing a level that its past self already beat, and in beating it, changed.

This is the “clear cache at 1pm” problem. The agent learns to clear the cache when disk usage hits 80%. The cache is now clear. The next time the agent faces the container, the state is different because of what it learned to do last time. The distribution didn’t shift on its own; the agent’s own competence shifted it. You can’t build a map of a river that changes course every time you step in it.

This is exactly the regime where the assumptions of standard continual learning collapse. Past data describes worlds the agent itself destroyed. A cumulative replay buffer doesn’t preserve knowledge; it preserves contradictions. A sliding window doesn’t discard knowledge; it discards ghosts.

So the test is simple. Two agents, same base model (Qwen 2.5 7B Instruct), same environment, same selection rule, same training objective. The only difference is how they carry data forward:

• Terese uses a cumulative buffer. She trains on everything: every successful trajectory from every generation. Terese v2 starts with 5,000 rows. By v13, she has 65,000. She is the standard CL instinct made concrete: never forget, always accumulate.

• Miri uses a sliding window of K = 3. She trains only on the three most recent generations. Miri v6 trains on data from v3, v4, and v5. Miri v7 trains on data from v4, v5, and v6. The oldest generation is discarded every time. She is the alternative: forget aggressively, keep only what’s current.

The question is not which agent learns faster. It is what happens to each when the agent’s own actions make past data describe destroyed worlds. Does Terese’s growing archive of incompatible realities help or hurt? Does Miri’s enforced amnesia leave her stranded, or does it protect her from the ghosts?

Results

Terese plateaued. Miri didn’t.

That’s the headline, but the mechanism is what matters. The two agents share the same base model, the same environment, the same selection rule, the same training objective. The only difference is how they carry data forward. And that difference explains everything about what happens next.

Terese’s failure mode. The cumulative buffer averages over incompatible disk states. Early generations describe containers full of log files; later generations describe containers where those logs were already deleted. Training on both simultaneously produces a policy that tries to clear caches that no longer exist and preserve files that were already removed. The agent wastes compute on ghosts.

Efficiency drops as the buffer grows, not because of overfitting, but because the data contradicts itself. Each generation adds trajectories that are locally valid but globally incompatible with the trajectories already stored. The buffer swells from 5,000 rows to 65,000, and the policy gets worse, not better. This isn’t catastrophic forgetting. It’s catastrophic averaging. The agent isn’t forgetting old skills; it’s being trained to execute actions that are mutually exclusive across the environments it has inhabited.

Miri’s sustained improvement. The K=3 sliding window discards data from destroyed worlds. It trains only on trajectories generated in the current state of the environment. If a strategy from three generations ago still works, the agent re-derives it from fresh interaction and it re-enters the window naturally. If it doesn’t work because the agent itself destroyed the conditions that made it work, it never comes back.

The result: Miri v13 achieves over 50% of Terese v13’s peak performance, trained on roughly a quarter of the data. The sliding window doesn’t need more data to compensate. It needs less data, because the data it keeps actually describes the world the agent is in. Monotonic improvement across generations, no plateau, no collapse, precisely because Miri never tries to learn worlds that no longer exist.

The concrete signal. The semantic maps make this visible. Terese’s maps show her attempting variants of “clear cache” and “delete log” that are irrelevant to the current disk state: strategies that worked in v2 but describe nothing real in v10. The clusters spread and dilute as the buffer accumulates contradictions. Miri’s maps stay concentrated on actions that are valid now. The clusters tighten with each generation as the agent prunes what doesn’t work and consolidates what does.

This is the core finding: the cumulative buffer isn’t adding signal. It’s adding noise from the past. And in a self-induced non-stationary system, the past is the noisiest source you have.

Why the Results Are Interesting

The continual learning (CL) literature is obsessed with “catastrophic forgetting.” The field is built on the premise that the primary failure mode is loss: the agent learns task B, and unlearns task A. Methods like Elastic Weight Consolidation (EWC) and replay buffers are designed to act as bulwarks against this erosion, preserving the past at all costs.

But the Miri result suggests the field has misdiagnosed the patient. In an open-ended loop where the agent destroys the world it learns from, the past isn’t gold; it’s toxic. Terese didn’t fail because she forgot how to solve earlier problems. She failed because she remembered them too well. She hoarded trajectories from v2 through v13, averaging over disk states that no longer exist. The cumulative buffer didn’t preserve knowledge; it preserved contradictions. This isn’t catastrophic forgetting. It is catastrophic remembering. The policy was poisoned by observations from worlds the agent itself had already demolished.

This flips the script on sliding windows. As documented in our sliding window SFT vs best-of-N study, standard CL often treats a sliding window as a naive hack, a concession to memory constraints that sacrifices performance on earlier tasks. It is “data destruction,” a dirty word in a field obsessed with retention. But Miri proves that when earlier tasks describe environments that are fundamentally incompatible with the current one, discarding them isn’t a sacrifice; it’s a correction. The window doesn’t destroy knowledge; it curates reality. It enforces a strict epistemic hygiene: if a strategy from three generations ago is still valid, the agent will re-derive it in the current environment. If it isn’t valid, it is garbage. The window simply takes out the trash.

This distinction matters because it exposes a feedback loop that standard RL and CL methods are blind to. Researchers treat non-stationarity as an external nuisance: the weather changes, the road curves, and the agent scrambles to adapt. But self-induced non-stationarity is different. The drift rate isn’t random; it is proportional to the agent’s competence. The faster the agent learns to clear the cache, the faster the cache disappears. Better learning leads to faster environment change, which leads to harder learning. Competence accelerates the obsolescence of the very data that produced it. Standard methods, designed to track external drift, miss this entirely. They try to anchor the agent to a past that the agent has already rendered obsolete.

What Might Explain Them

The Miri result is striking, but the mechanism behind it is what matters. Why does a sliding window of three generations outperform a cumulative archive of thirteen? Why does forgetting help when the field treats it as the enemy? The answer lies in three facts about how the agent and environment interact.

The non-Markovian state-of-the-disk. Standard reinforcement learning assumes a Markov Decision Process: state \({S}_{t}\) leads to action \({A}_{t}\) leads to reward \({R}_{t}\). The state at time \(t+1\) depends only on the state and action at time \(t\). This assumption works when the environment is a fixed game board, such as Atari, chess, or a maze.

But when the agent’s actions permanently alter the environment, the Markov property breaks. The state at \(t+1\) is not just a function of \({S}_{t}\) and \({A}_{t}\); it is a function of the entire history of policies the agent has executed. The “state” isn’t the observation: the container’s current disk usage, the list of processes, or the file tree. The state is the accumulated residue of every action the agent has ever taken. The agent deleted /var/log/syslog in generation v2, and that deletion persists through v13. The state space itself is a moving target, and the agent’s own competence is what moves it.

A cumulative replay buffer tries to learn a policy \(\pi (a|s)\) that averages over all possible disk states. While external agent memory stores past notes as passive prompt context, parameter fine-tuning forces the policy to internalize these transitions directly into weights. It mixes trajectories from v2, where the cache is full, with trajectories from v10, where the cache is already empty. The resulting policy is muddy: correct for no particular state, diluted by the ghosts of environments past. Terese didn’t fail because she forgot. She failed because she remembered too much, averaging over mutually incompatible realities.

Why the window tracks the drift. The sliding window doesn’t approximate the derivative of environment change; that claim is unproven and poetic. What it does is simpler and more honest: it trains on data generated under the current policy-environment coupling.

If the agent’s last three generations of actions produced the current disk state, then data from those generations describes the world as it actually is. Older data describes the world as it was before the agent changed it. The window enforces a strict recency discipline: show me you can still do it. If a strategy from three generations ago is still valid, the agent will re-derive it in the current environment, and the data will re-enter the window naturally. If it isn’t valid because the agent itself destroyed the conditions that made it work, it never comes back.

This is why Miri improves monotonically. She isn’t carrying the baggage of v2 through v12. She is learning from the world as it exists right now, which is the only world that matters for the next action.

The environment as its own replay buffer. The deepest insight is that the sliding window doesn’t need to store valid old strategies; it needs the agent to re-execute them. The loop is closed through the environment, not through the buffer.

If a strategy learned in generation v3 is still valid in v6, the agent will rediscover it because the environment rewards it, and the successful trajectory re-enters the window. The environment acts as a generative replay buffer: it produces fresh instances of the same problem, and the agent produces fresh solutions. Old data doesn’t need to be hoarded; it needs to be re-validated. The window simply enforces that validation step.

This is the opposite of standard continual learning, which treats the past as a repository of timeless truth. In self-induced non-stationarity, the past is a repository of destroyed worlds. The sliding window doesn’t preserve knowledge; it curates reality. It keeps what works now and discards what worked then. And in a system where the agent’s own success makes the past obsolete, that is not a bug; it is the only way to survive.

Theoretical & Speculative Interest

The results raise questions that the current literature isn’t equipped to answer. That’s not a criticism of the field; it’s a consequence of the reframe. Once you stop treating non-stationarity as exogenous, the standard tools and standard analyses don’t quite apply.

Task-free CL is the nearest literature, and it still misses the point. Task-free continual learning relaxes the assumption that task boundaries are known or even discrete. The agent faces a stream of shifting data and must adapt without being told when one task ends and another begins. This is closer to the NSL setting than Split CIFAR, but it still carries a hidden assumption: the distribution shifts exogenously, then holds stationary long enough for the agent to consolidate. There are stable interludes.

Self-induced non-stationarity has no interludes. The distribution is always drifting because the agent is always acting. The drift rate is coupled to the agent’s competence: the better it gets, the faster the world changes. There is no “between tasks” because the agent’s own learning is the event that renders the current task obsolete.

The relevant literature is actually closer to continual reinforcement learning without replay buffers

, where agents must learn in non-stationary environments without storing past experiences. That work uses generative models to hallucinate past data, a kind of synthetic rehearsal. But NSL agents don’t need to hallucinate. They re-encounter the actual environment. If a past strategy is still valid, the environment produces fresh instances of the problem; the agent produces fresh solutions; the data re-enters the window. The loop is closed through reality, not through a generative model. The environment is the replay buffer.

The feedback loop as a design constraint. Here is a property that the standard CL framework cannot express: the drift rate is proportional to the learning speed. The faster the agent learns to clear the cache, the faster the cache disappears. Better learning accelerates the obsolescence of the very data that produced it.

This creates a coupling that most methods ignore. Any CL technique that accelerates learning also accelerates the obsolescence of its own training data. You cannot speed up adaptation without speeding up the world’s retaliation. This suggests an equilibrium: the optimal window size K is the one where the data refresh rate matches the environment drift rate. Too small a window and the agent can’t learn stable structure; it doesn’t see enough examples to consolidate anything. Too large a window and it trains on stale worlds, averaging over disk states that no longer exist. K = 3 worked in the NSL experiments, but that is an empirical result in one environment. The general principle (matching the refresh rate to the drift rate) is unexplored. We don’t yet have the theory to predict K from the dynamics of the agent-environment coupling.

Self-induced non-stationarity as a feature, not a bug. Most agent training tries to minimize the agent’s impact on the environment. Evaluations are sandboxed. Environments are reset between episodes. The agent faces a pristine world every time, and the goal is to measure how well it performs without leaving a mark.

But real agents change things. That is the point of acting. A deployed system administrator doesn’t return the server to its original state after each intervention. A deployed medical AI doesn’t reset the patient. The agent’s actions persist, and the environment it faces tomorrow is different because of what it did today. If we want agents that survive in the wild, we need training procedures that assume the environment will be different after deployment because the agent was deployed.

The sliding window is a first step toward training for that reality. It doesn’t try to freeze the world. It doesn’t try to preserve the past. It accepts that the agent will change its environment, and it trains the agent to track the world it is creating, generation by generation, with no recourse to the ghosts of environments past.

Try It Yourself

The Negative Space Learning codebase makes these experiments reproducible with config changes, not code changes. The key parameter is orchestration.training_window_size, the number of recent generation shards included in each training round. The entry point is scripts/run_train_loop.py.

Reproduce the cumulative-buffer vs. sliding-window comparison. Create two configs from the same base (e.g., config/config-memory-cleanup-msagent.toml). In the Terese config, set training_window_size equal to num_generations (this makes the window grow with each round, accumulating all past data). In the Miri config, set training_window_size = 3. Fix the environment seed via generation.variation_random_seed so both agents face the same container variations in the same order. Run:

uv run python scripts/run_train_loop.py --config config/terese.toml
uv run python scripts/run_train_loop.py --config config/miri.toml

Compare the semantic maps and per-generation success rates in the output directories. The divergence should appear within 5–8 generations: the cumulative buffer starts averaging over incompatible disk states, and Terese’s per-generation metric flattens or degrades while Miri’s continues climbing.

Vary the window size K. Run five sweeps: training_window_size = 1 (total amnesia, where each generation trains only on its own data), 3 (Miri), 5, 10, and a value equal to num_generations (Terese). Plot per-generation performance across all five. The hypothesis: performance peaks at small K and degrades as K grows, the opposite of the standard CL prediction where larger replay buffers should help. If you see that inversion, you have direct evidence that the environment drift is self-induced. If the drift were exogenous, larger K would stabilize learning. When the agent is causing the drift, larger K poisons it.

Inspect the buffer contradiction. After 10 generations with the cumulative buffer, go into the generation output directories and sample trajectories from generation 2 versus generation 10. The data lives under generation_output_dir/<run_id>/generation_<N>/. Look at the actions each trajectory describes: early generations will contain strategies like clearing full caches and deleting large log files; later generations will contain strategies that assume those caches and logs are already gone. Check whether the disk states they describe are mutually compatible: they won’t be. Training on both simultaneously is the mechanism of failure. You can verify this directly by computing the loss on held-out trajectories from each era: a model trained on the full cumulative buffer will have high loss on both early and late trajectories, while a model trained on the K=3 window will have low loss on recent trajectories and appropriately high loss on obsolete ones, which is the correct answer because those earlier trajectories describe worlds that no longer exist.