TL;DR

  • Keeping a model learning after training ends usually means writing to an external memory file, not updating the model’s own weights - the way a fruit fly’s brain may encode navigation directly into synaptic strength rather than a sustained loop of neural firing. Closing that gap is the harder, weight-level version of recursive self-improvement (RSI).
  • More training does not reliably mean a better model: Continuously training a model during inference risks catastrophic forgetting (the model overwrites what it already knew).
  • Three real approaches exist for updating weights continuously: reinforcement learning (narrow, moves only a small targeted slice), LoRAs (discardable bolt-on weight blocks), and differentiable plasticity (a second system decides which of the first system’s weights are editable - the most elegant option, but the slowest to train).

Key Takeaways

  • More training does not reliably mean a better model. Continuous training during inference risks catastrophic forgetting - the model overwriting what it already knew - so the less of the model you touch on each pass, the less you risk breaking.
  • Reinforcement learning is the most common way to update weights continuously. Useful, but not a path to broad continuous learning on its own.
  • LoRAs and differentiable plasticity are the two alternatives. LoRAs are discardable bolt-on weight blocks - blunter, but scalable; differentiable plasticity trains a second system to decide which of the first system’s weights to edit - more elegant, but far slower to train.

Continual learning, done for real: three approaches that work - and why a billion sacrificed fruit flies got there first

Peter Wang laid out the puzzle well. A fruit fly that wanders off from a drop of sugar water in the dark can always find its way back - a feat called path integration, which requires keeping a running sum of every step it’s taken. The neurons that report each step are well characterised, built on decades of work by Larry Abbott, Gaby Maimon, Vivek Jayaraman and Barbara Webb. The neurons that actually add those steps up have never been found.

Screenshot of two tweets by Peter Wang: the first shows a fly navigating a simulated environment toward a banana, captioned that the fly connectome reveals neurons enabling navigation via fast synaptic weight updates rather than neural activations; the second lays out the path integration puzzle in full
Peter Wang (@BrainsAndTennis): “This is fast-weight continual learning in a fly, something current LLMs don't do!” - the tweets this piece grew out of

There are two ways a brain could hold a sum like that. The usual answer is that some neurons hold it in their activations, sustained by exciting each other in a loop tuned precisely enough that the signal neither fades nor blows up - which is also how RNNs and LLMs hold state. The other answer is that nothing keeps firing at all: each step gets encoded straight into synaptic strength, so the sum of all the weight changes is the sum of the journey.

A few papers had proposed the fly works this way, but nobody had pointed to candidate neurons for it - until now. Four previously unremarkable neuron types, hΔH, hΔA, hΔI and hΔG, turn out to have every ingredient the second theory needs: input from the step-reporting neurons, velocity-sensitive dopamine input that could gate memory writing, and a reward-sensitive octopamine neuron that could reset the weights when the fly reaches food. Simulations confirm it’s a viable candidate for path integration.

That distinction - hold the memory in activations, or hold it in the weights themselves - is the same one that separates a model that remembers things by writing to a memory.md file between sessions from one that’s actually updating its own weights as it goes. If you’re interested in recursive self-improvement (RSI), that should excite you.

Not because we’re about to run Skynet on a fly, but because it looks like a working biological example of the harder, weight-level version of the mechanism. Evolution ran that search for a billion generations so the fly wouldn’t have to think about it.

What would it take for a model to close a version of the same loop? We’ve found three basic ways to attack this. Here’s how each one holds up.

The problem with more training

The main obstacle to continuous learning - and hence to recursive self-improvement at the weight level - is that more training doesn’t necessarily make a model better. Grokking, where a model overtrains, appears to plateau, and then suddenly gets dramatically better, is one example of this non-linearity between training and skill (we’ve written before about what looks like a T-schema forming inside a grokking model). In practice that shows up as a model’s test scores improving as it grows familiar with a dataset, then getting worse as it overfixates and loses the ability to answer questions it hasn’t memorised, then improving sharply again once it works out how to encode the relationships between categories rather than just the surface facts.

You can see a related “double descent” phenomenon in a typical training/test loss curve: with training, the model gets okay, then not okay, then spectacular.

Line chart of training loss and test loss plotted against training steps on a log scale: training loss falls steadily and flattens out, while test loss climbs sharply before collapsing back down to meet it
Double descent: test loss gets worse before it gets dramatically better

If you keep training a model while it’s inferencing, every additional hour raises the odds you’ll run into catastrophic forgetting - the model starts forgetting load-bearing old material in order to learn new material. It’s a bit of a crapshoot: the more chances you take, the likelier it is that your number comes up. To avoid it you need to limit the proportion of the model that gets retrained on any given pass. The less you touch, the less you break.

Reinforcement learning’s narrow lane

Currently the most popular way to do continuous weight updates is reinforcement learning (RL), which covers a bunch of techniques that mostly involve getting the model to try to solve a specific set of problems and grading its answers on the fly.

This limits weight changes in a few ways. First, the answers are all generated by the model itself, so you’re not asking it to do anything it didn’t already have buried in its heart of hearts - you’re just shifting the likelihood it chooses one route over another. Second, because you’re training it in a specific direction, the resulting changes concentrate on the specific set of weights related to the behaviour you actually care about, and those changes end up moving broadly in a similar direction.

Third, developers add thorough automated checks to stop the model straying too far from its base or checkpoint. That’s not always intellectually satisfying, but it works.

We’ve written elsewhere about why RL specifically breaks down once you leave curated settings for uncurated, mass-market ones. The point here is narrower: even when it works, RL only ever moves a small, targeted slice of the model.

LoRAs: weights you can throw away

If you don’t like RL, you have other options. You can do what we do and build LoRAs - blocks of weights you bolt onto the side of a model that influence its outputs without touching the model’s original weights at all. If the LoRA turns out to perform badly, you just throw it away.

Diagram of a LoRA: an input X passes through both the frozen pre-trained weights W and a smaller trainable weight update ΔW, and the two resulting outputs are summed to produce the final output W.x + ΔW.x
A LoRA bolts a small, trainable ΔW block onto the side of a model’s frozen weights W - discard the block and the original model is untouched

LoRAs have fallen out of fashion lately, seen as a) unscalable - you can’t just keep adding new LoRAs forever - and b) un-beautiful. Which they are, but. We mitigate the scaling problem by using sliding-window LoRAs that train continuously based on the model’s own experience, which gets us safe adaptability and the ability to learn things outside the model’s original distribution. (Full details in Survival is the Only Reward.) The model chooses what it wants to experience, and hence what to learn, so coherence is provided by the entity doing the experiencing rather than by a policy imposed on it from outside.

It’s still a fairly blunt instrument, though, and if we could target weight updates within the base model directly, we would.

Differentiable plasticity: the most elegant option

Which brings us to a third type of continual learning, by far the most elegant. It involves training a second system to decide which of the first system’s weights are editable on any given pass - differentiable plasticity. Within a set-up like this, System 2’s only job is to work out which of System 1’s weights need to be modified to assimilate any given learning experience.

There are various examples. Schmidhuber (who else?) has produced a couple of neat variants: a self-referential weight matrix that learns to modify itself, and a broader survey connecting fast weight programming and linear transformers to neurobiology.

That survey covers the same territory Schlag, Irie and Schmidhuber cover from the other direction, showing that linear transformers are secretly fast weight programmers too.

Peter Wang picked up that last thread in the replies with two links rather than one, and they’re worth keeping separate because they’re doing different jobs. The first, just above, is the theoretical grounding - the formal proof that a linear-attention mechanism and a fast-weight controller are the same thing.

The second is where that equivalence shows up in production: Gated Delta Networks (Yang, Kautz and Hatamizadeh) combines a gating mechanism with a delta-rule weight update to improve on Mamba2, and by Wang’s account this delta-rule mechanism now underlies Qwen3.5 and other modern LLM architectures. As he put it: “in some sense, ICL in SSMs = fast weight updates.” That’s not a hedge - he named the specific mechanism and the specific model line it’s already shipping in.

Abbott himself - from the original fly thread - built a version where a single network organises itself into bits that learn and bits that tell other bits when to learn: meta-learned synaptic plasticity and memory addressing for continual familiarity detection, with Danil Tyulmankov and Guangyu Robert Yang.

Even closer to the fly model is Backpropamine, which has the catchiest title in recent ML history and describes a bio-inspired simulation of the kind of chemical neuromodulation fruit flies actually use.

DeepMind’s current major RSI vehicle, Titans, uses a similar principle, with different components all instructing one another on how to evolve. For my money this is the least beautiful of the lot - it specifies so much by hand - but up to now it’s the most scalable, and after all, who am I to demand beauty from this process?

The main issue with these systems is that you need time and data to train the second layer. If training a regular LLM is the equivalent of educating a kid from birth to age 20, this is the equivalent of evolving that kid’s cerebellum over tens of millions of years.

Why piggyback on a fly

And that, finally, is why the fly paper is exciting: because the fly already did this. A billion drosophila sacrificed themselves in horrible ways to find the brain configuration for wayfinding back to a drop of sugar water in the dark. Working this out from scratch - even in AI-facilitated search over the relevant program space - would be a frustrating, time-consuming chore. If we can piggyback on a fruit fly to get there quicker instead, that’s a big deal.

So: congratulations on making it to the end of this one before the singularity does.