LoRA Stacking vs Fresh Adapters: Chaining Without Drift

TL;DR: Merging \(\text{LoRA}_1\) into the base before training \(\text{LoRA}_2\) looks like the obvious way to run a multi-generation fine-tuning loop, but LoRA stacking turns the base into a moving target. Each merge into a 4-bit base dequantises, adds the delta and requantises, and it bakes the previous adapter’s intruder dimensions into the reference for the next one, so representation drift compounds with every generation. Training each generation’s adapter fresh from the frozen base, on a sliding window of the successful trajectories the previous adapter collected, chains generations through data instead of weights: every adapter stays one rank-\(r\) delta from the same base, forgetting is bounded by the window, and rollback is a file swap.

What LoRA Stacking Does to the Base Over Many Generations

When we began working on continual learning loops, a core part of recursive self-improvement, we had several options: train an adapter on the first batch of experience, fold it into the base weights with merge_and_unload(), then train the next adapter on top. LoRA stacking of this kind feels natural because it mirrors how checkpoints accumulate in pretraining. The next thing we considered was to keep one adapter checkpoint and resume training it every generation. Both choices carry state forward in the weights, and both produce representation drift: the hidden states and token distributions move a little further from the pretrained model with every generation, in directions no single adapter chose.

Drift matters more in a loop than in a one-off fine-tune because each generation trains on data the previous generation produced. A shift in the model’s representations also shifts the distribution of its own future training data, so an error introduced at generation 3 still shapes what generation 10 sees. That feedback is the self-induced distribution shift that makes continual loops harder than sequential task benchmarks.

Where Sequential LoRA Merging Compounds Error

Requantisation Noise in 4-Bit Merges

Start with quantisation. A QLoRA setup, from Dettmers et al. (2023) introducing 4-bit NormalFloat QLoRA, stores the base in 4-bit NormalFloat and trains the adapter in 16-bit, which is how most teams fit a Qwen 3.6 or Gemma 4 base on one GPU. Merging BA into that base is not an addition.

PEFT dequantises the weight, adds the delta and quantises again, and its merge path for 4-bit linear layers warns that the merged module may generate differently because of rounding errors. Do that once and you have a few bits of noise. Do it every generation and the base at generation 10 carries ten rounds of requantisation that no adapter was ever trained to correct.

The quantization-aware QA-LoRA design by Xu et al. (2023) removes the rounding loss for one case: it constrains the adapter so that the merge folds into the quantiser’s zero points, and the merged model stays in INT4 with unchanged outputs. That is built for deploying a single merged model. It still rewrites the base once per merge, and it needs its own group-wise constraint during training.

Structural Drift and Intruder Dimensions

The second source of drift is structural. Shuttleworth et al. (2024) analyzed LoRA intruder dimensions by comparing the singular value decompositions of LoRA-updated and fully fine-tuned weight matrices and found that LoRA introduces intruder dimensions: new singular vectors with large singular values that are nearly orthogonal to the pretrained ones.

Models carrying them match the target task but model the pretraining distribution worse and adapt less robustly when tasks arrive sequentially. In a stacked loop the next adapter is trained against a base that already contains the previous adapter’s intruder dimensions, and then adds its own.

Keeping a single adapter and resuming it each generation avoids requantisation but not accumulation. The A and B matrices carry every past generation’s gradient history and optimiser moments, nothing bounds how far BA moves from zero over time, and there is no clean way to ask what the model would do without generation 4’s data in it. The history is in the weights, where you cannot edit it.

Chaining Fresh LoRA Adapters Through Data

The alternative changes what the chain is made of. Generation \(t\) collects trajectories with the frozen base plus \(\text{LoRA}_{t-1}\), keeps the ones the environment marks as successes, and then trains \(\text{LoRA}_t\) from scratch on the frozen base over a sliding window of the last few generations’ successes. The adapters are chained through data: \(\text{LoRA}_{t-1}\) decides which experiences exist, \(\text{LoRA}_t\) learns from them, and the base both sit on never changes. It is the pattern behind nightly adapters, and it is what makes a loop cheap to run for many generations.

Four Bounds on Representation Drift

This bounds representation drift for four reasons:

• One delta, always from the same origin. Every generation is one rank-\(r\) update from the pretrained weights. The distance from the base is bounded by what one adapter can express, not by the sum of \(t\) adapters, and nothing ever requantises the base.
• Intruder dimensions die with their adapter. \(\text{LoRA}_t\) is trained against the pretrained singular structure, not against whatever \(\text{LoRA}_{t-1}\) introduced.
• Forgetting is a setting, not an accident. What the model has absorbed is exactly the window. Drop a generation from the window and it is absent from the next adapter, which turns catastrophic forgetting into a budget you choose.
• Comparison and rollback are file operations. Every adapter is a delta from the same base, so evaluating generation 6 against generation 3 means loading two files, and reverting a bad generation is a swap. Rollback stays a safety net; the window does the real work.

Minimal Chaining Loop in PEFT

In PEFT the loop is short. The base is loaded once, each generation’s actor is the base with the previous adapter attached, and the learner is a new get_peft_model call on the same base object (Hugging Face PEFT repository):

from peft import LoraConfig, PeftModel, get_peft_model
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

base = AutoModelForCausalLM.from_pretrained(
    BASE_ID, quantization_config=BitsAndBytesConfig(load_in_4bit=True))
cfg = LoraConfig(r=16, lora_alpha=32, target_modules="all-linear")
WINDOW = 3  # generations of successes the next adapter trains on

for t in range(1, GENERATIONS + 1):
    # Collect with last generation's adapter on the frozen base.
    actor = PeftModel.from_pretrained(base, f"adapters/gen-{t-1}") if t > 1 else base
    save_rows(collect_successes(actor, env), f"data/gen-{t}.jsonl")
    if t > 1:
        base = actor.unload()  # remove LoRA_{t-1}; nothing is merged

    # Train a fresh adapter from the same base on the window.
    window = load_rows(f"data/gen-{k}.jsonl" for k in range(max(1, t - WINDOW + 1), t + 1))
    learner = get_peft_model(base, cfg)  # new A and B, base untouched
    train_sft(learner, window)
    learner.save_pretrained(f"adapters/gen-{t}")
    base = learner.unload()

The cost is real. Each generation retrains from zero, so the adapter relearns whatever the window carries, and a skill whose examples have left the window is gone unless the agent reproduces it. That is the trade: a bounded, inspectable memory in exchange for the open-ended accumulation that LoRA stacking offers.

When Merging and Composition Are the Right Tools

Chain of LoRA: Residual Learning on Fixed Targets

Merge-and-reinitialise exists for good reasons, and a continual loop is not the setting it was designed for. Biderman et al. (2024) measured full fine-tuning perturbations at 10 to 100 times the rank of typical LoRA configurations, which explains part of why a single adapter learns less on a fixed target.

The Chain of LoRA approach by Xia, Qin and Hazan (2024) answers that directly: merge each adapter into the frozen weights, start a new one to learn the residual, and let the rank of the total update grow with each link. It outperforms a single LoRA across seven tasks on OPT and Llama 2 with no extra memory. When the goal is to close the gap to full fine-tuning on one dataset, stacking is the right tool. A continual loop wants the opposite: a bounded, replaceable update on a distribution that the model itself keeps changing.

Task Composition vs Generational Stacking

Adapter merging across tasks is also a different operation from stacking across generations. PEFT’s add_weighted_adapter combines several task adapters into one with TIES task vector merging or DARE, which trim the deltas and resolve sign conflicts before summing. Each of those adapters was trained from the same base on its own task, and the merge is a one-off that can be redone from the originals at any time.

The orthogonal subspace O-LoRA architecture by Wang et al. (2023) goes further for tasks that arrive in sequence: it keeps each task’s adapter in a subspace orthogonal to the earlier ones, so nothing is merged at all. Every one of these methods shares the property that matters here: the base stays fixed and every adapter is a delta from it. LoRA stacking across generations is the one design that gives that property up.

FAQs

Is LoRA stacking the same as Chain of LoRA?

Mechanically yes: both merge an adapter into the weights and train a new one on top. The difference is the objective. Chain of LoRA merges within one fixed dataset to raise the rank of the final update. Stacking across generations merges on self-generated, shifting data, where each merge also moves the reference for the next adapter.

Can I merge an adapter into a 4-bit base without changing its outputs?

Not with a plain merge: PEFT dequantises, adds and requantises, and warns that generations may differ. QA-LoRA avoids the loss by folding the adapter into the quantiser’s zero points, but it needs its own training constraint. Merging into a 16-bit master copy and requantising once is the usual compromise for deployment.

How large should the sliding window be?

Large enough that the adapter sees every skill the agent still needs, and small enough that stale experience from an earlier policy does not dominate. Two or three generations is a sensible start. Measure held-out task performance after every generation, older tasks included, and widen the window only when something you still need drops out.

Does a fresh adapter forget what the earlier ones learned?

It never sees their weights, only their data. Any skill still represented in the window is relearned from successful trajectories, which is cheaper than it sounds because the previous adapter produced them. Skills that fall out of the window are gone by design, which is the bounded forgetting the window exists to control.

Why not keep training the same LoRA checkpoint across generations?

Continuing to train one LoRA checkpoint avoids 4-bit requantisation, yet it still accumulates every past generation’s gradient updates and optimiser moments inside the adapter matrices. Over time the adapter drifts unconstrained from zero, and stale habits from early policies remain baked into the weights with no way to drop them selectively.

When should you use TIES or DARE adapter merging instead?

Use TIES or DARE adapter composition when you have independent task adapters that were each trained once from the same frozen base model and want to combine them for deployment. Because the original task adapters stay untouched on disk, you can recompute or reweight that merge at any time without compounding error.