LoRA Stacking vs Fresh Adapters: Chaining Without Drift

Fri, 06 Nov 2026 09:00:00 +0800

Sequential merging turns the base into a moving target; fresh adapters chain through data and keep the base constant

TL;DR: Merging \(\text{LoRA}_1\) into the base before training \(\text{LoRA}_2\) looks like the obvious way to run a multi-generation fine-tuning loop, but LoRA stacking turns the base into a moving target. Each merge into a 4-bit base dequantises, adds the delta and requantises, and it bakes the previous adapter’s intruder dimensions into the reference for the next one, so representation drift compounds with every generation. Training each generation’s adapter fresh from the frozen base, on a sliding window of the successful trajectories the previous adapter collected, chains generations through data instead of weights: every adapter stays one rank-\(r\) delta from the same base, forgetting is bounded by the window, and rollback is a file swap.

PEFT on Eigenform Continuous Learning Tool Tips

LoRA Stacking vs Fresh Adapters: Chaining Without Drift