TL;DR
- Continual learning in current models usually means writing to an external memory file, not updating the model’s own weights. A fruit fly’s brain may instead encode navigation directly into synaptic strength, with no sustained loop of neural firing. Closing that gap is the harder, weight-level version of recursive self-improvement (RSI).
- More training does not reliably mean a better model. Continuously training a model during inference risks catastrophic forgetting: the model overwrites what it already knew.
- Three real approaches exist for updating weights continuously: reinforcement learning (narrow, moves only a small targeted slice), LoRAs (discardable bolt-on weight blocks), and differentiable plasticity (a second system decides which of the first system’s weights are editable; the most elegant option, but the slowest to train).
Key Takeaways
- More training does not reliably mean a better model. Continuous training during inference risks catastrophic forgetting, the model overwriting what it already knew, so the less of the model you touch on each pass, the less you risk breaking.
- Reinforcement learning is the most common way to update weights continuously. Useful, but not a path to broad continual learning on its own.
- LoRAs and differentiable plasticity are the two alternatives. LoRAs are discardable bolt-on weight blocks: blunter, but scalable. Differentiable plasticity trains a second system to decide which of the first system’s weights to edit: more elegant, but far slower to train.
This is continual learning done for real: three approaches that work, and why a billion sacrificed fruit flies got there first.
What is continual learning?
Continual learning is a machine learning model’s ability to acquire new knowledge or skills from a continuous stream of data and experience after initial training, without losing what it already knows. As framed in the continual learning survey by Liyuan Wang and colleagues, it addresses how an adaptive system accumulates capabilities over time rather than freezing parameters at deployment.
In current models, continuity usually means writing notes to an external memory file between sessions, leaving underlying weights unchanged. The harder and more consequential challenge is updating weights continuously during inference. Closing that gap represents the necessary step toward weight-level recursive self-improvement (RSI), moving beyond external memory files to models that modify their own parameters.
What does a fruit fly show about continual learning?
Peter Wang laid out the puzzle well. A fruit fly that wanders off from a drop of sugar water in the dark can always find its way back. The feat is called path integration, and it requires keeping a running sum of every step the fly has taken.
The neurons that report each step are well characterised, built on decades of work by Larry Abbott, Gaby Maimon, Vivek Jayaraman and Barbara Webb. The neurons that actually add those steps up have never been found.
Two ways to hold a running sum
There are two ways a brain could hold a sum like that. The usual answer is that some neurons hold it in their activations, sustained by exciting each other in a loop. That loop has to be tuned precisely enough that the signal neither fades nor blows up, which is also how RNNs and LLMs hold state. The other answer is that nothing keeps firing at all: each step gets encoded straight into synaptic strength, so the sum of all the weight changes is the sum of the journey.
A few papers had proposed the fly works this way, but nobody had pointed to candidate neurons for it until now. Four previously unremarkable neuron types, hΔH, hΔA, hΔI and hΔG, turn out to have every ingredient the second theory needs.
They get input from the step-reporting neurons, velocity-sensitive dopamine input that could gate memory writing, and a reward-sensitive octopamine neuron that could reset the weights when the fly reaches food. Simulations confirm it’s a viable candidate for path integration.
That distinction (hold the memory in activations, or hold it in the weights themselves) is the same one that separates two kinds of model. One remembers things by writing to a memory.md file between sessions. The other is actually updating its own weights as it goes. If you are interested in recursive self-improvement (RSI), that should excite you.
Not because we’re about to run Skynet on a fly, but because it looks like a working biological example of the harder, weight-level version of the mechanism. Evolution ran that search for a billion generations so the fly wouldn’t have to think about it.
What would it take for an artificial model to close a version of that same weight-level loop? Examining what learning in weights unlocks makes the incentive obvious.
What are the advantages of continual learning?
The main advantage of continual learning is that a model can adapt from its own experience without retraining from scratch. Rather than relying solely on external memory files between sessions, learning from its own data by encoding changes directly into weights provides a direct path toward continuous adaptation.
Moving from external notes to internal parameters represents the foundational step toward weight-level recursive self-improvement. The practical value of this shift depends on how updates are managed: reinforcement learning concentrates modifications around curated objectives, touching only a small targeted slice of the model; Low-Rank Adaptation (LoRA) provides discardable and scalable add-on blocks; and differentiable plasticity introduces selective weight editing guided by a secondary controller.
Yet all of these gains depend on a fragile condition: they only hold if the model does not forget what it previously learned.
What are the challenges of continual learning?
The main obstacle to continual learning, and hence to recursive self-improvement at the weight level, is that more training doesn’t necessarily make a model better. Grokking is one example of this non-linearity between training and skill: a model overtrains, appears to plateau, and then suddenly gets dramatically better. (We’ve written before about what looks like a T-schema forming inside a grokking model.) In practice, a model’s test scores improve as it grows familiar with a dataset. Then they get worse as it overfixates and loses the ability to answer questions it hasn’t memorised. Finally they improve sharply again once it works out how to encode the relationships between categories rather than just the surface facts.
You can see a related “double descent” phenomenon in a typical training/test loss curve: with training, the model gets okay, then not okay, then spectacular.
If you keep training a model while it’s inferencing, every additional hour raises the odds you’ll run into catastrophic forgetting: the model starts forgetting load-bearing old material in order to learn new material. It’s a bit of a crapshoot. The more chances you take, the likelier it is that your number comes up. To avoid it you need to limit the proportion of the model that gets retrained on any given pass. The less you touch, the less you break.
What is catastrophic forgetting?
Catastrophic forgetting occurs when a neural network overwrites what it learned earlier after being trained on new material. In continual learning, this degradation represents the primary hazard of updating parameters during inference: adjusting shared weights to absorb recent inputs disrupts configurations that earlier capabilities relied upon.
The article’s working precaution is straightforward: retrain a smaller share of the model on each pass, reducing the surface area where things break. The three approaches implement that principle through distinct mechanisms rather than equivalent fixes. Reinforcement learning narrows updates toward targeted task behaviour. LoRAs add discardable low-rank blocks while leaving the base model unchanged. Differentiable plasticity trains an independent second system to select editable weights.
What are the main continual learning techniques and approaches?
Three practical continual learning approaches allow models to update weights after initial training: reinforcement learning, Low-Rank Adaptation (LoRA), and differentiable plasticity. In the wider research literature, one survey by Wang et al. groups some methods into regularization, replay, and architecture-based categories. That taxonomy serves only as one example grouping rather than an exhaustive framework, and it does not map onto the three mechanisms examined here. This article focuses strictly on practical techniques that update parameters continuously during inference.
These three continual learning techniques manage parameter updates through different architectural trade-offs. Reinforcement learning restricts updates to a small, task-specific slice of base weights, steering behavior toward curated objectives without altering broader capabilities. In contrast, LoRA continual learning leaves base parameters untouched, bolting on discardable low-rank blocks that can be removed if performance degrades. Differentiable plasticity uses a secondary meta-learning system to decide which base weights are editable on any given pass. Comparing these types of continual learning shows how each balances update precision against structural complexity.
The table below summarizes these three continual learning approaches, contrasting what each mechanism updates against its operational strengths and structural limitations.
| Approach | What it updates | Strength | Limitation |
|---|---|---|---|
| Reinforcement learning | A narrow, targeted slice of base model weights | Narrows gradient updates toward task-relevant behaviours, touching less of the model | Confined to narrow, curated objectives; does not provide a route to broad continual adaptation |
| LoRAs (Low-Rank Adaptation) | Discardable bolt-on weight blocks ($\Delta W$) outside frozen base weights | Highly scalable when paired with sliding windows; defective blocks can be discarded without altering base weights | A blunt instrument that cannot directly rewrite or fine-tune representations inside the base model |
| Differentiable plasticity | Selective base weights determined by a trained second system | Allows selective parameter adjustments rather than unconstrained global updates | Slow to train |
Can reinforcement learning enable continual learning?
Currently the most popular way to do continuous weight updates is reinforcement learning (RL). Applying reinforcement learning continual learning covers a bunch of techniques that mostly involve getting the model to try to solve a specific set of problems and grading its answers on the fly.
This limits weight changes in a few ways. First, the answers are all generated by the model itself, so you’re not asking it to do anything it didn’t already have buried in its heart of hearts. You’re just shifting the likelihood it chooses one route over another. Second, because you’re training it in a specific direction, the resulting changes concentrate on the weights related to the behaviour you actually care about. Those changes end up moving broadly in a similar direction.
Third, developers add thorough automated checks to stop the model straying too far from its base or checkpoint. That’s not always intellectually satisfying, but it works.
We’ve written elsewhere about why RL specifically breaks down once you leave curated settings for uncurated, mass-market ones. The point here is narrower: even when it works, RL only ever moves a small, targeted slice of the model.
How can LoRAs support continual learning?
If you don’t like RL, you have other options. You can do what we do and build LoRAs: blocks of weights you bolt onto the side of a model that influence its outputs without touching the model’s original weights at all. If the LoRA turns out to perform badly, you just throw it away.
LoRAs have fallen out of fashion lately, seen as a) unscalable, since you can’t just keep adding new LoRAs forever, and b) un-beautiful. Which they are, but. We mitigate the scaling problem by using sliding-window LoRAs that train continuously based on the model’s own experience. In our setup, LoRA continual learning provides safe adaptability and the ability to learn things outside the model’s original distribution. (Full details in Survival is the Only Reward.) The model chooses what it wants to experience, and hence what to learn, so coherence is provided by the entity doing the experiencing rather than by a policy imposed on it from outside.
It’s still a fairly blunt instrument, though, and if we could target weight updates within the base model directly, we would.
What is differentiable plasticity in continual learning?
Which brings us to a third type of continual learning, by far the most elegant. It involves training a second system to decide which of the first system’s weights are editable on any given pass: differentiable plasticity. Within a set-up like this, System 2’s only job is to work out which of System 1’s weights need to be modified to assimilate any given learning experience.
What are fast weight programmers?
There are various examples. Schmidhuber (who else?) has produced a couple of neat variants with Kazuki Irie and Imanol Schlag: a self-referential weight matrix that learns to modify itself (arXiv:2202.05780), and a broader survey connecting fast weight programming and linear transformers to neurobiology.
That survey covers the same territory Schlag, Irie and Schmidhuber cover from the other direction, showing that linear transformers are secretly fast weight programmers too.
How do Gated Delta Networks use fast weights in production?
Peter Wang picked up that last thread in the replies with two links rather than one, and they are worth keeping separate because they perform different jobs. The first, just above, is the theoretical grounding: the formal proof that a linear-attention mechanism and a fast-weight controller are the same thing.
The second link points to empirical benchmarking. The paper on Gated Delta Networks (Yang, Kautz and Hatamizadeh) evaluates architecture performance, demonstrating that combining a gating mechanism with a delta-rule update improves synthetic associative recall and language modeling benchmarks over Mamba2. The paper itself establishes benchmark results rather than production deployment. In the replies, Wang offered his own interpretation, arguing that this delta-rule mechanism informs modern production models such as Qwen3.5: “in some sense, ICL in SSMs = fast weight updates.” On Wang’s reading, fast weight programming is quietly finding its way into deployed architectures.
What is neuromodulation in neural networks?
In neuromodulation, from Abbott to Backpropamine, artificial neural networks emulate biological chemical signalling to regulate plasticity. Abbott himself, from the original fly thread, built a version where a single network organises itself into bits that learn and bits that tell other bits when to learn: meta-learned synaptic plasticity and memory addressing for continual familiarity detection, with Danil Tyulmankov and Guangyu Robert Yang.
Even closer to the fly model is Backpropamine, which has the catchiest title in recent ML history. It describes a bio-inspired simulation of the kind of chemical neuromodulation fruit flies actually use.
What does Titans cost to train? The second system problem
DeepMind’s current major RSI vehicle, Titans, uses a similar principle, with different components all instructing one another on how to evolve. For my money this is the least beautiful of the lot, since it specifies so much by hand. But up to now it’s the most scalable, and after all, who am I to demand beauty from this process?
The main issue with these systems is that you need time and data to train the second layer. If training a regular LLM is the equivalent of educating a kid from birth to age 20, this is the equivalent of evolving that kid’s cerebellum over tens of millions of years.
Why piggyback on a fly?
And that, finally, is why the fly paper is exciting: because the fly already did this. A billion drosophila sacrificed themselves in horrible ways to find the brain configuration for wayfinding back to a drop of sugar water in the dark. Working this out from scratch, even in AI-facilitated search over the relevant program space, would be a frustrating, time-consuming chore. If we can piggyback on a fruit fly to get there quicker instead, that’s a big deal for continual learning.
Where is continual learning heading? The future of continual learning
Where continual learning goes next hinges on moving beyond external memory files toward genuine weight-level adaptation. Updating weights directly remains the critical bridge to practical self-improving AI. Yet existing approaches leave open trade-offs: sliding-window LoRAs offer scalable adaptation but remain structurally blunt, while differentiable plasticity targets specific weights at the cost of substantial training time for the second system.
Biological models offer a possible head start instead of searching from scratch. Rather than unguided exploration across architectural spaces, the fruit fly’s candidate neuron types (hΔH, hΔA, hΔI, and hΔG) provide a concrete hypothesis for how gating and reward signals might coordinate synaptic changes.
So: congratulations on making it to the end of this one before the singularity does.
FAQs
What is continual learning in practice?
Continual learning means acquiring new capabilities and adapting from ongoing experience after initial training while retaining earlier knowledge. While many current systems only simulate continuity by writing session history into external memory files or context windows, continual learning in practice alters the model’s own parameters. Updating weights continuously represents a step toward weight-level recursive self-improvement, though turning static models into self-improving systems is not an established outcome.
Why is catastrophic forgetting the primary risk during inference?
When standard training runs continuously while a model is inferencing, gradient updates across shared network parameters can overwrite the weight configurations that support earlier tasks. Because the network reuses internal representations, modifying weights on fresh inputs risks degrading previously established competencies. Retraining only a smaller share of the architecture on each pass serves as a practical precaution to reduce that interference.
What are the main continual learning approaches?
The article examines three approaches for updating weights continuously. Reinforcement learning shifts a narrow, targeted slice of base parameters toward specific problem-solving behaviours. LoRA bolts on discardable low-rank weight matrices beside frozen base weights, keeping updates modular. Differentiable plasticity trains a secondary meta-learning system to decide which primary weights are editable on a given pass. Each manages parameter updates differently.
How does differentiable plasticity work?
Differentiable plasticity uses a two-system structure where a secondary meta-learning network determines which primary network weights should change during an experience. Instead of allowing unconstrained gradient shifts across the whole model, System 2 evaluates incoming inputs and selects editable weights in System 1. This bio-inspired approach enables selective internal parameter adjustments, though training the secondary gating system requires significant time and data.
Can LoRAs be used for continual learning?
Yes. While standard LoRAs are static fixtures, sliding-window LoRAs enable ongoing adaptation from a model’s own operational experience. Trainable low-rank matrices attach beside frozen base weights, so the core parameters remain untouched. If a particular LoRA performs poorly or degrades behaviour, developers can simply discard it without damaging the base model. However, LoRAs remain a relatively blunt instrument compared to direct weight edits.
What are the practical advantages of continual learning?
Continual learning allows an AI system to adapt directly from its ongoing experience without having to retrain entirely from scratch. Encoding modifications into internal weights rather than external memory files represents the foundational step toward weight-level recursive self-improvement. Depending on the technique, practitioners gain specific advantages: narrow task targeting in reinforcement learning, discardable modularity in LoRAs, or selective weight editing in differentiable plasticity.
How is continual learning different from using external memory?
Using external memory means a model records notes or session context to an external file between interactions, leaving its underlying weights completely unchanged. In contrast, continual learning modifies the model’s own parameters during or after inference. Updating internal weights allows the system to assimilate new experience directly into its structure, rather than relying on separate files to simulate continuity.
