Catastrophic forgetting in continual fine-tuning: a model cube trained over three rounds loses older skill tiles as new ones arrive, beside four techniques that reduce it: a LoRA adapter, a sliding window, rehearsal data and a KL anchor to the base model
Each round of fine-tuning can erase older skills

TL;DR: Naive sequential fine-tuning pulls neural network weights toward the newest dataset and overwrites parameters that encoded older capabilities, causing catastrophic forgetting. Narrow or cherry-picked self-generated buffers accelerate that collapse: in our 13-generation container study, retraining only on top-scoring past runs fell back to baseline. Practical continual fine-tuning combines several partial defences: training fresh nightly LoRA adapters from scratch on a frozen base model each round, bounding history with a recency sliding window, mixing rehearsal data, or tethering token distributions with a KL divergence anchor, backed by per-round held-out evaluations and adapter rollback as a safety net.

Why Can’t You Just Keep Fine-Tuning?

Once you accept that AI agent memory isn’t learning, the obvious way to make a model learn continuously is to fine-tune it again whenever new data arrives: this week’s successful agent runs, next week’s, and so on. Each round, the model gets better at the newest data. It also gets worse at things it used to do well, including general abilities nobody was trying to change.

This is not an edge case. A continual instruction-tuning study found forgetting across LLMs from 1B to 7B parameters, and it grew more severe as models scaled within that range.

Training is also non-linear. A model can regress before it improves, so every extra round is another chance to break something that worked. The less of the model each round touches, the less it can overwrite: the design principle behind most continual learning methods.

Inside a recursive self-improvement loop where an agent periodically trains on successful episodes collected from its own environment, forgetting shows up on every round, so the training architecture has to manage it explicitly.

What Is Catastrophic Forgetting?

Catastrophic forgetting is the sharp loss of previously learned abilities when a neural network is updated on new data. Gradient steps for the new task shift weights that also supported earlier tasks, and without structural or distributional constraints, accuracy on older capabilities can drop abruptly rather than decay gradually.

McCloskey and Cohen documented this interference in connectionist networks in 1989. Most modern mitigations follow one of three routes: penalise movement on important weights, restrict trainable updates to a small subspace, or keep representative data in the active buffer.

Why Continual Fine-Tuning Forgets

Three mechanisms drive forgetting in a continual fine-tuning loop, while a fourth point is often overlooked: bounded forgetting is sometimes beneficial.

Every Update Moves Shared Weights

In full-parameter fine-tuning, every weight coordinate is free to move. Pre-trained skills are superposed across those same weights, so optimising exclusively for this week’s batch shifts layers that earlier reasoning and formatting skills depended on.

Narrow Data Narrows the Model

When a model trains on its own rollouts, each round’s dataset reflects what the previous policy favoured. In our Negative Space Learning experiments, three lineages of Qwen 2.5 7B were retrained on their own surviving episodes: the lineage trained only on its three highest-scoring past runs collapsed, the one accumulating all past runs plateaued, and the one trained on a sliding window of recent runs kept improving.

The Newest Data Dominates

Rows from the latest round often outnumber or out-specify examples from earlier states, and uniform cross-entropy treats every row equally. Without deliberate windowing or rehearsal, updates follow whichever distribution dominates the latest batch.

Bounded Forgetting Helps in Shifting Environments

Forgetting core competence is a failure, yet discarding obsolete tactics is not. In the same container study, structured recency forgetting prevented niche overfitting: strategies that stopped freeing storage aged out of the window, while strategies that still worked kept reappearing in recent rollouts. The engineering goal is controlled forgetting, not infinite retention.

Ways to Prevent Catastrophic Forgetting, and Their Limits

No single technique eliminates forgetting; practical pipelines layer complementary defences:

TechniqueHow it limits forgettingCostIn NSL
LoRA adaptersBase weights stay frozen; only a low-rank update is trainedLower single-round capacity than full SFTAlways: QLoRA on a frozen base model
Restart from the base each roundDrift cannot compound across stacked adaptersAdapter reflects only its current windowEvery generation fits a fresh adapter
Sliding window of recent dataOld rows age out on a fixed scheduleDrops skills absent from recent windowstraining_window_size, default 3
Rehearsal dataGeneral text in the mix keeps broad abilities activeExtra rows and step timerehearsal_dataset, configurable
KL anchor to the base modelPenalises divergence from base-model token probabilitiesReference pass overheadASFT with asft_kl_weight > 0
Weight-importance penalties (EWC)Stiffens weights critical to earlier tasksPer-parameter Fisher storageNot used

LoRA Adapters: Train Less, Forget Less

A LoRA adapter trains a compact low-rank update while base parameters remain frozen. Studying how LoRA forgets less, Biderman et al. (2024) showed that restricting updates to a low-rank subspace preserves out-of-domain base performance substantially better than full fine-tuning, weight decay, or dropout, at the price of lower peak plasticity on large distribution jumps.

Restarting a Fresh Adapter From the Base Model

NSL never stacks adapters sequentially. Every generation loads the frozen base checkpoint and fits a fresh LoRA adapter from scratch, preventing approximation drift from compounding across iterations. Earlier generations influence the current policy strictly through the surviving episodes inside the active window.

Sliding Windows and Rehearsal Mixes

A sliding window retains the last \(K\) rounds of validated trajectories and drops older shards (training_window_size = 3 by default in NSL). When rehearsal_dataset and rehearsal_rows_per_epoch are enabled, the trainer mixes general pre-cutoff text alongside task trajectories so broad language fluency stays exercised.

Anchoring Output Distributions to the Base Policy

Anchored SFT (ASFT) pairs confidence-weighted token loss with a forward KL penalty against a reference policy. With LoRA, the reference distribution comes directly from toggling the adapter off on the frozen base model (asft_kl_weight > 0; at 0, ASFT reduces to unanchored DFT).

Parameter-space regularisers take a different path: Elastic Weight Consolidation (EWC) estimates diagonal Fisher importance per weight coordinate to slow movement on parameters critical to prior tasks.

How NSL Layers These Protections

In our research on approximate policy-free online learning, agents periodically collect episodes that pass an environmental survival gate, and two distinct training components work together: modified SFT (adjusting token loss and optional KL anchoring) and fresh QLoRA adapters trained over a three-generation sliding window. Rehearsal mixes and asft_kl_weight tighten retention when a workload demands stronger base stability, while archiving prior adapters provides a rollback safety net.

That combination manages the plasticity-stability trade-off explicitly: the window sets the forgetting horizon, low-rank adapters cap single-step displacement, and base-model restarts stop errors from compounding.

How to Measure Catastrophic Forgetting in Your Own Loop

With self-generated data, forgetting also interacts with distributional narrowing, where recursive training on generated data erodes tail diversity. Treat each defence above as an independent experimental lever:

  1. Freeze a held-out evaluation suite before round one: combine tasks from early states with an out-of-domain coding or reasoning benchmark never included in training.
  2. Compare buffer retention schedules: evaluate cumulative history, recency sliding windows, and peak-score curation with base model and hyperparameters held constant.
  3. Log four metrics after every generation: old-skill retention, target-task gain, trajectory diversity, and regression frequency against the prior checkpoint.
  4. Ablate one knob at a time: vary training_window_size, lora_rank (default 32), rehearsal_rows_per_epoch, and asft_kl_weight separately so effects stay attributable.
  5. Run at least ten generations: late collapses (such as top-K curation diverging around generation 12) rarely appear in two or three rounds.

Operational Details to Watch For

  • Evaluate held-out general benchmarks after every generation, applying the same monitoring discipline that catches production model drift.
  • Archive each after_generation_<N> adapter on disk so reverting a bad update is a configuration switch rather than a retraining run.
  • Size lora_rank to the task gap: very small ranks protect base capabilities well yet can underfit novel tool schemas.
  • Check sequence truncation logs when using rehearsal datasets so rows exceeding max_seq_length are not silently dropped.

FAQs

Does LoRA prevent catastrophic forgetting completely?

No. Freezing base weights and constraining updates to a low-rank subspace reduces out-of-domain forgetting compared to full fine-tuning, yet sequential updates or narrow self-generated datasets can still collapse adapter behaviour unless paired with fresh base restarts, bounded windows, or KL anchoring.

How do you reduce catastrophic forgetting in continual LLM fine-tuning?

Combine structural and data-level defences rather than relying on one fix: fit fresh LoRA adapters from the frozen base model each round, curate recent valid data via a sliding window plus a small rehearsal set, and add a forward-KL anchor when vocabulary drift appears.

Why is restarting a fresh LoRA from the base model important?

Stacking adapters or continuing gradient updates on the same LoRA across many rounds lets approximation errors and narrow biases compound cumulatively. Fitting a fresh LoRA adapter from scratch on the frozen base checkpoint each generation ensures earlier rounds influence the policy strictly through the surviving data window.

Is forgetting always harmful in continual agent learning?

No. Losing foundational reasoning or coding syntax is harmful, whereas dropping obsolete environment-specific tactics helps agents adapt when states change. In our 13-generation container runs, structured forgetting via a three-generation sliding window outperformed both full historical accumulation and top-score curation.

How many fine-tuning rounds can an LLM run before it forgets?

There is no universal round limit because retention depends on dataset narrowness, learning rate, adapter rank, and KL regularisation. Track a fixed held-out benchmark alongside your target task after every round so regressions trigger an immediate adapter rollback safety net.