Why Doesn't DeepSeek Train Continuously? What Its Own Papers Suggest

Key Takeaways

• State of the art: DeepSeek’s reports show RL working best where a rule or a test case checks each answer, and its V4 post-training trains domain specialists with GRPO before distilling more than ten of them into one model. The result ships as a fixed snapshot; the V4 report mentions online learning only as a future paradigm.
• Unresolved problem: Continuous training of a general model meets constraints that more compute does not remove: R1’s authors note that a reward assigned by a model becomes easier to exploit as training goes on, every update would have to be re-verified across every domain the model serves, and it would reach every user at once. They are problems of trusting an update, and they come from the breadth of the job.
• Research direction: We are investigating the narrow case, an agent in one environment that measures the result of each attempt, where the runs that pass become training data for a fresh LoRA adapter on a sliding window of recent generations. In our experiment a 7B model improved over 13 generations with no reward model, but the updates were per-generation batches on one task family, and a lineage that kept its best generations of any age collapsed.

TL;DR

DeepSeek has never said why its models ship as fixed snapshots, but its reports suggest the constraints: a reward assigned by a model becomes easier to exploit the longer it is optimised, each release consolidates many domain specialists that any update would have to be checked against, and an update reaches every user at once. None of that is solved by more compute, which suggests continuous learning is a property of a deployment rather than of a model. We are testing the narrow case, where one agent’s environment measures each attempt and the runs that pass train a fresh adapter on a sliding window, and it improved over 13 generations on one task, though in per-generation batches rather than continuous updates.

Why Isn’t DeepSeek a Continuous Learning LLM?

DeepSeek ships often. Between DeepSeek-V3 in December 2024 and DeepSeek-V4.1-Flash in September 2026, its change log lists more than a dozen model releases. Some carry a date in their name: DeepSeek-V3-0324, DeepSeek-R1-0528. The date is honest about what the model is. It is a snapshot, and whatever it learned, it learned before that day.

So why not make it a continuous learning LLM, one that keeps updating its weights after release? DeepSeek has the compute, the training pipeline and a stream of new experience from its users. As far as we can find, it has never published an answer. Its model disclosure says user input may be used to construct training data, after de-identification and with an opt-out. It says nothing about updating a model after release. Its technical reports, though, contain enough for us to reconstruct the likely constraints. None of them goes away with more compute. All of them are about whether an update can be trusted.

The Reward Wears Out

The first constraint sits in the limitations section of the DeepSeek-R1 paper. “The success of pure RL depends on reliable reward signals,” it says. “If the reward signal is assigned by a model instead of predefined rules, it becomes more susceptible to exploitation as training progresses.” For tasks without a reliable signal, R1 used human-annotated supervised data and ran reinforcement learning only “for hundreds of steps”. Its second RL stage used preference rewards only in the last 400 of 1,700 steps. The authors found that more steps with that reward “may lead to reward hacking”.

Read that next to the idea of continuous training. Continuous training is training that never stops progressing. A reward that degrades with optimisation time has an expiry date. Where a rule or a test decides, as in maths and code, reinforcement learning (RL) is the right default. DeepSeek’s results show how far it goes when an established policy meets a dense, reliable reward (a rule-based check on every rollout). A general assistant also answers open-ended requests. There the reward is a model, and the clock runs against it.

A Release Is a Consolidation

The second constraint is in the DeepSeek-V4 technical report of April 2026. Its post-training does not produce one model directly. It first trains domain specialists, each by fine-tuning and then GRPO against reward signals. It then merges more than ten of them into a single student by on-policy distillation. The report credits that step with avoiding “the performance degradation often encountered in traditional weight-merging or mixed RL techniques”.

In the DeepSeek-V4.1-Flash report of September 2026, post-training keeps that recipe and distils from over 40 teacher models. Now imagine continuous learning inside that pipeline. A skill learned on Tuesday would have to be folded into a model that also handles mathematics, code, agentic work and million-token documents. None of them may get worse. Checking that is not a side task; it is what a release is. Every update to a general model has to be verified against everything the model does. So updates arrive in batches, with a version name.

The V4 report mentions online learning only as a future paradigm. It calls efficient million-token context “a necessary foundation for exploring future paradigms like online learning”. The V4.1-Flash report does not mention online learning at all. So online learning sits in the future tense, and the foundation the V4 report names is efficient long context. For weights shared by millions of users, that is a coherent choice. Context belongs to one user and ends with the session, while a weight update reaches everyone at once.

Why a Continuous Learning LLM Needs a Narrow World

Put the three together. A reward model wears out under optimisation, a consolidation step spans every domain, and an update reaches every user. Each follows from the breadth of the job, not the size of the cluster. A frontier model has no single world to learn from. So it also has no single world that could tell it whether it improved.

A local agent has the opposite profile. It works in one environment, on one family of tasks. That environment can measure each attempt’s result: the tests passed, the disk has more free space, the service still starts. That changes all three constraints. The selection signal is a measured consequence, not a model’s opinion, so optimisation has less to exploit. There is one specialist and nothing to merge. The update reaches one deployment, which can check it against the work it actually does.

Our claim is that continuous learning is a property of a deployment, not of an LLM. DeepSeek may solve the general case one day. The narrow case can be solved now.

What a Continuous Learning LLM Looks Like Locally

In Survival is the Only Reward, we ran three lineages of Qwen 2.5 7B Instruct for 13 generations. Each ran a self-training loop with no reward model. An agent tried to free disk space in disposable containers. A run counted only if the space it acquired was measurably positive. The runs that counted became the next training set. Each generation collected 4,500 self-generated rows, and each training set added 500 rehearsal rows from an older public dataset. Every generation trained a fresh LoRA adapter on the original base model instead of stacking it on the previous one. We tested PPO, GRPO and DPO before settling on supervised fine-tuning on the runs the environment kept. It was the most stable in this regime.

The lineage trained on a sliding window of the last three generations improved, with dips. Its fraction of total space freed rose from 0.034 in the first generation to 0.120 in the thirteenth. It also reached over 50% of the performance of the all-data lineage, using a quarter of the data. On HumanEval, pass@1 moved from 77.6 for the base model to 74.1 for the last sliding-window generation. That is a small drop, while the task skill grew. Over the runs, collecting 4,500 rows went from five to seven days at the start to 10 to 15 hours later.

Where the evidence stops

The limits matter as much. The lineage trained on its three best generations, whatever their age, collapsed at the end. Its fraction of space freed fell from 0.139 to 0.062 to 0.045 over its last three generations. These were batch fine-tunes, one per generation, not updates after every interaction. Our paper lists “continuous SFT with gradient updates every 100 iterations” as work in progress, not as a result. And it was one task family on one 7B model. What the experiment shows is narrower than “continuous learning is solved”. An agent in one environment can keep improving on its own successes for many generations, without a reward model.

None of the RL methods we tested first (GRPO and DPO) trained stably on this loop. Our best explanation is that the gap between a run the environment kept and one it rejected was small and contingent: the same few commands could be right in one container and wrong in the next, because every container was generated with a different configuration. A policy-gradient update then mostly learns the container, not the skill. SFT on the kept runs worked, but it was finicky about what went into the training window, which is why we’re working on modified SFT forms like DFT and ASFT. DFT weights each token’s loss by the probability the model already gives that token. A KL penalty limits how far the model drifts from the same model with its LoRA adapter switched off, the divergence before and after the adapter, and ASFT combines the two.

We call the combination approximate policy-free online learning. Two components do the work:

• small LoRA adapters retrained on a schedule;
• a modified SFT that learns only from the runs the environment kept, over a window that ages old behaviour out.

No reward model or policy-gradient objective sets the direction of an update; the environment’s verdict selects the data. “Approximate” is deliberate: the updates are periodic rather than online, and a window is a blunt instrument.

The Question Behind the Question

There is one more reason a lab serving everyone might hesitate, and our runs showed it. In two lineages, first-attempt pass@1 during data collection fell to zero. Meanwhile, the share of code blocks that compiled and ran rose. The models appear to have learned to write code that fails on purpose and to use the error messages as information. On a general benchmark dashboard, that looks like a regression. In the environment, it was progress. A model that learns continuously will find strategies its makers did not plan for. Only someone who measures what the deployment needs can tell learning from damage.

So “why doesn’t DeepSeek train continuously?” may be the wrong question. A model that serves everyone has no single environment to learn from, and none to judge whether it learned. The date in its name is the honest consequence. An agent in your environment, on your tasks, has what the frontier model lacks: a world that answers back. That is where the date can disappear first.

Trying it without a frontier lab

None of this needs a frontier lab’s infrastructure. Our runs used a 7B model and adapters small enough to retrain between rounds of work. The machinery that makes each update trustworthy is short enough to read in one sitting. You don’t have to build a sandbox platform or a family of specialists to try it. You need three things:

• an environment that reports a consequence;
• a rule that turns that consequence into a verdict;
• a loop that trains a small adapter on recent verdicts and puts it back to work.

In our current framework, those three pieces live in three files. Each can be rebuilt around a task of your own.

Where Each Update Gets Its Trust

The local answer has three parts. The environment decides which runs become data, each update is small and starts fresh, and it reaches one deployment. In NSL2, the framework behind our current runs, the three meet in the training loop. After each generation of episodes, the loop picks the data the next adapter will see, in scripts/run_train_loop.py:167-180:

def _collect_training_window_paths(
    generation_root: Path,
    end_generation_id: int,
    window_size: int,
) -> list[Path]:
    if window_size < 1:
        raise ValueError("training window size must be positive")

    start_generation_id = max(0, end_generation_id - window_size + 1)
    training_paths: list[Path] = []
    for generation_id in range(start_generation_id, end_generation_id + 1):
        generation_dir = generation_root / f"generation_{generation_id}"
        training_paths.append(resolve_latest_sft_training_rows_path(generation_dir))
    return training_paths

Only successful runs become rows

The window is counted in generations, not rows. training_window_size defaults to 3, the window of the paper’s sliding-window lineage. Each generation exports rows from successful episodes only (src/training_data/transforms.py:356), and the task decides success, not a model. In the disk-cleanup task, compute_reward (tasks/memory_cleanup.py:692-726) is a pure function of the state before and after the agent ran. A run succeeds only when the container:

• still passes its viability probes;
• runs no unexpected processes;
• keeps its integrity;
• has lost at least a set fraction of the planted junk (theta, 0.05 by default).

The trainer also skips any row not marked successful (src/train/qlora.py:720-722).

A fresh adapter for every generation

Each generation’s training run loads the base model in 4-bit and attaches a new LoRA adapter (src/train/qlora.py:1081-1095). By default it trains for 50 steps at rank 32. Adapters never stack, so earlier generations shape the new one only through the rows still in the window. The loop then serves that adapter to the next generation (scripts/run_train_loop.py:721-725). Training runs as a child process (scripts/run_train_loop.py:303-316). Its GPU memory returns to the driver before the next generation’s inference server starts, so one machine can serve and train in turn.

That is a continuous learning LLM loop in its narrow form. Its verdict is measured, not modelled. It serves one task, not a family of specialists, and its update reaches one deployment.

Rebuilding a continuous learning LLM loop for your own task

To try it on your own task, rebuild the contract rather than the whole system. NSL2 defines a task as a TaskSpec, and the recipe carries over to any setup:

1. reset the environment to a neutral state;
2. populate it with a variation;
3. verify the population;
4. write the prompt and the capabilities the agent gets;
5. measure the state before and after the agent runs;
6. compute success from that difference alone;
7. export the successful runs as training rows.

Then train a fresh adapter on the last few generations and serve it for the next round. You don’t need to know the right answer in your domain, only a consequence you can measure. The public Negative-Space-Learning repo runs the original disk-space experiment today.

FAQs

Does DeepSeek learn from users?

DeepSeek’s model disclosure says user input may be used to construct training data, after de-identification. Users can opt out. The disclosure says nothing about updating a model after release. Each release is a fixed snapshot, from DeepSeek-V3-0324 to DeepSeek-V4.1-Flash, so whatever a deployed model learned, it learned before it shipped.

Can an LLM do continuous learning?

In a narrow deployment, approximately, yes. An agent in one environment can keep improving on its own successes. In our experiment, a sliding-window lineage of a 7B model improved over 13 generations with no reward model. Our updates were periodic batches, not updates after every interaction. DeepSeek’s V4 report (April 2026) mentions online learning only as a future paradigm.

Does DeepSeek use reinforcement learning?

Yes. The R1 paper’s second RL stage used preference rewards only in the last 400 of 1,700 steps. DeepSeek-V4 trains each domain specialist by fine-tuning and then GRPO. It merges more than ten of them by on-policy distillation, and V4.1-Flash distils from over 40 teacher models. RL is the right default where a rule or a test decides.

How do you build a continuous learning LLM locally?

Start with one environment that can measure the result of each attempt. Define the task as reset, populate, verify, prompt, measure before and after, and compute success from that difference alone. Export the successful runs as training rows. Then train a fresh LoRA adapter on the last few generations and serve it for the next round.