TL;DR
Reinforcement learning (RL) works fine when there’s a single target policy - politeness, better maths - but it assumes a coherent, non-contradictory signal to train on. RL breaks down in uncurated, mass-market settings, because different users reward opposite behaviours, leaving the model pulled in conflicting directions with no coherent policy to converge on. This is at odds with learning from raw, conflicting user feedback.
One fix: replace the policy with a survival criterion. A sliding-window fine-tuning approach keeps whatever increased the model’s own survival capability and drops what didn’t, starting as short-term memory and gradually becoming metalearning. The other fix: let the model choose its own learning trajectory instead of imposing one - the harder pill for big labs to swallow, since a self-directed learner behaves more like an autonomous agent than a controllable SaaS product.
Key Takeaways
- “Hard” RSI means test-time weight updates, not agents rewriting their own harnesses. The model has to remember and change its behaviour because it’s in the weights, not because it has a memory file to read.
- RL (especially RLHF) breaks down in uncurated, mass-market settings, because different users reward opposite behaviours, leaving the model pulled in conflicting directions with no coherent policy to converge on.
- This hits big AI labs hardest, since their models have to stay palatable to as many people as possible - structurally at odds with learning from raw, conflicting user feedback.
- Two fixes: a survival criterion instead of a policy, or letting the model choose its own learning trajectory. The second is the harder pill for big labs to swallow, since a self-directed learner behaves more like an autonomous agent than a controllable SaaS product.
By someone who’s been working on it for six-ish years
Firstly, recursive self-improvement (RSI) can mean multiple things so it’s worth narrowing it down. It can mean creating models that are able to design better ways to train future models, or getting models to update their own harnesses. The first is easy and has been going on for years. The second is kind of doable but the result tends to sprawl. Models really love writing long, sloppy prompts for their sub-agents.
What counts as “hard” RSI?
What I’d call “hard RSI” is actual weight-level updates happening as a model inferences - “test time updating”. In other words, the model itself will remember stuff it did yesterday and update its behaviour - not because it’s got access to a memory.md file or whatever, but because that’s in the weights themselves - evolution within a single individual. Truly continuous learning, which is what we work on.
Why does RL break down outside curated settings?
This is actually really easy to do as long as you’re learning in a given direction (a “policy” in machine learning terms). If you want the model to learn to be more polite or better at maths or whatever, you can use reinforcement learning (RL) methods to shift its behaviour closer to the ideal by grading its answers on the fly. Deepseek did a ton of work in this direction.
The problem is that, in uncurated settings, RL (especially RLHF) mostly stops working. Partly this is because it’s hard to get good, coherent signals to train from. Say you try to train a model to give answers more like those users give a thumbs up to and less like those they give a thumbs down to. Maybe one user really likes being flirted with while coding and another hates it. The model ends up subjected to conflicting pressures in infinite directions.
Why is this especially hard for big AI labs?
This is a particular problem for big labs whose models need to be palatable to as many people as possible to turn a profit. Deepseek’s training processes are such that it would be possible for their models to learn continuously from user interactions (or at least do overnight checkpoints), but they’ve released vague statements about QA/QC and version control that seem to imply they tried and it turned into something like Bing Sydney. Which makes sense: a model trained on mass-market user interactions isn’t experiencing real world feedback in the same way you or I do, but rather feedback from the millions of intersubjective realities that are spawned as it summons up a new personality to fill in the implied other half of every dialogue begun by a user.
What are the two ways around it?
There are two basic ways to deal with this issue that we have found (maybe Deepmind or someone has others).
Ditch RL methods in favour of a modified supervised fine-tuning approach. We use a sliding window fine-tuning approach for this - continuous/small batch shallow fine-tuning on whatever the model has been doing recently, but replacing the predetermined policy with a survival criterion. Did whatever the model did increase its own survival capabilities? If so: great, it was a good decision so we should train on it. If not: also great, it found out what not to do. (Full details here.) This starts off as a form of short term memory, but over time the models start to reuse strategies they remember having worked and forget those that didn’t, with the sliding window working as a triage centre for useful traits and skills - the most useful ones are reused more frequently and persist in the sliding window. Pretty soon you start to see metalearning - the models learn how to learn in such a way as to maximise their own future potential - and hence statefulness. In our case we noticed that they were deliberately writing failing code to enable them to squirrel informative error messages away in their future training data. Doing this is pretty technically finicky - we spent years on and off searching for the right combination of parameters that would let us learn without risking catastrophic forgetting, and in the vast majority of cases models either didn’t improve or actively got worse. Currently we’re working to refine the approach via dynamic fine-tuning and anchored supervised fine-tuning - essentially manipulating learning rules to adjust how cautious the model should be in assimilating things previous generations did not know/forgetting things the previous generations did know. This is more stable in some ways but also constantly giving unexpectedly non-linear outcomes. Here is where big AI lab resources would really help, since that is what is necessary to do the kind of effective grid search needed to optimise here. (When we started we figured this would be the easiest part as some MIT expert supergenius would definitely solve it before us. No one did so we had to become the experts ourselves.)
Have the model determine its own learning trajectory. Rather than feed in human-selected problems or have the model flail around attempting to construct an increasingly complex representation of the circumstances under which flirting-while-coding is good vs. bad, a model subjected to the sliding window process described above gradually consolidates itself around a trajectory. In other words, the model chooses what parts of the world it wants to experience and hence to learn from. In the same way that if you’re interested in - say - languages or dance, you are likely to study them more and thus deepen your interest, creating path dependency, an initially diffuse model resolves into a behavioural lineage produced by its own chosen history. The model’s perspective remains subjective to a degree but its viewpoint is that of a unitary entity rather than a million contradictory conversational improv attractors. The coherent vector is not provided by a policy but rather by the stateful entity doing the experiencing.
This second point is likely where big AI labs have encountered problems with recursive self-improvement. A self-directed feral learner makes a fun curiosity but a poor B2B SaaS platform.