What Are Elastic Weights? EWC vs. LoRA in Continual Learning

Thu, 29 Oct 2026 14:35:00 +0900

EWC anchors individual parameter coordinates with virtual springs; LoRA isolates task adaptations to low-rank subspaces

Key Takeaways

Virtual springs on neural weights: Introduced by Kirkpatrick et al. (2017), Elastic Weight Consolidation (EWC) slows down catastrophic forgetting by anchoring each parameter \(\theta_i\) to its previous task solution \(\theta^*_{A,i}\) with a quadratic penalty weighted by diagonal Fisher Information \(F_i\).
Online and structured extensions: Classic EWC stores a new reference vector and Fisher tensor for every task. Online EWC (Schwarz et al., 2018) compresses these into a single running estimate, while Kronecker-factored Laplace approximations (Ritter et al., 2018) capture cross-weight correlations within layers.
Why classic EWC struggles on modern LLMs: Across billion-parameter Transformers, maintaining full-model Fisher states consumes heavy GPU memory, local Taylor approximations degrade as weights drift, and polysemantic neurons encode multiple entangled features inside the same parameter coordinates (Ning et al., 2026).
Subspace adapters and functional anchoring: In contemporary LLM pipelines, practitioners frequently pair or replace full-parameter EWC with Low-Rank Adaptation (LoRA), which confines updates to a low-dimensional subspace (Biderman et al., 2024; Zheng et al., 2026), alongside replay buffers or output-space KL regularisation.

TL;DR

Elastic Weight Consolidation (EWC) is one of the foundational algorithms in continual learning. Instead of allowing stochastic gradient descent to overwrite every neural network weight freely when training on a new task, EWC attaches a virtual spring to each weight whose stiffness matches how sensitive the previous task was to that coordinate. That simple physical idea works well on compact vision and reinforcement learning models, and its mathematical link to Bayesian posteriors and Fisher Information underpins a decade of continual learning research. At large language model scale, however, storing per-parameter curvature tensors and constraining raw weight coordinates runs into memory and geometry bottlenecks, pushing modern workflows toward low-rank adapters (LoRA), feature- or logit-space regularisation, and experience replay. Below is a clear, step-by-step walkthrough of how elastic weights work, where they shine, where they break down, and public PyTorch repositories you can run today.

Elastic Weight Consolidation on Eigenform Continuous Learning Tool Tips

What Are Elastic Weights? EWC vs. LoRA in Continual Learning

Key Takeaways

TL;DR