Is it mode collapse in LLMs, or incoherent training data?

TL;DR: When an iteratively trained language model climbs early on task benchmarks, oscillates for several rounds and then drops sharply over one or two generations, the failure can look like mode collapse from the outside. In our multi-generation sandbox experiments, we found that failure could just as likely be the result of an incoherent training signal rather than a loss of variety.

What mode collapse is and where it comes from in LLMs

In language modelling mode collapse refers to a post-training narrowing where a policy concentrates probability mass onto a small subset of output structures, reasoning templates, or action sequences. The model often stays fluent and scores well on single samples, yet repeated sampling yields near-duplicate trajectories.

This loss of behavioural variety is often attributed to reinforcement learning objectives, but recent post-training diversity research by Zhang et al. shows that the narrowing also has a data-level driver. Studying how post-training alignment reduces LLM diversity, Zhang et al. identify typicality bias in preference data, where annotators systematically favour familiar, conventional text over equally valid alternatives. When preference pairs already over-represent familiar phrasing, the training data selection step narrows the target distribution before the optimiser runs.

This problem becomes more complex, however, when the goal is test-time adaptation or online training. When a model enters a multi-generation loop where its own completions are scored, filtered, and fed back into subsequent fine-tuning rounds, the policy’s own bias towards high-probability modes and a preference or outcome filter can reinforce one another. Each generation samples mostly from modes that already carry high probability under the current policy, and when a preference filter favours conventional phrasing or an outcome filter keeps only the successful rollouts from that already narrowed pool, valid low-frequency strategies can drop out of the training data and narrow the policy further. Extending Zhang et al.’s finding from annotators to these loops is our extrapolation.

Practitioners often conflate this behavioural narrowing with recursive model collapse, yet the two phenomena have distinct causes and signatures. A data accumulation study by Gerstgrasser et al. shows that replacing real data with each generation’s synthetic data tends towards model collapse, a degenerative shift where approximation errors compound and erode tail coverage. Crucially, accumulating synthetic data alongside the original real data avoids model collapse.

Model collapse shows up as broad degradation in fluency and factual accuracy, whereas diversity loss usually leaves fluency and top-one accuracy intact while the repertoire of strategies contracts.

How to detect diversity loss and separate it from other failures

Single-sample metrics such as greedy accuracy or pass@1 cannot distinguish between a model that has expanded its reasoning capability and one that has simply concentrated probability mass onto a single existing solution path. When a post-training update shifts probability weight from secondary trajectories into the dominant mode, pass@1 often climbs over early iterations because the single most likely sample succeeds more frequently on standard test cases.

A sharper diagnostic is to evaluate pass@k across a wide range of k values at every training generation. In an RLVR reasoning boundary study, Yue et al. probe the reasoning capability boundary of RLVR-trained models with pass@k and show that RLVR-trained models perform well at small k while base models can match or exceed them at large k. That crossover at large k reveals a narrowed reasoning boundary: across many draws the post-trained model covers fewer solvable problems than the base model.

Alongside pass@k scaling, measuring output or strategy diversity directly across sampled rollouts shows whether the policy is converging onto one behavioural template. When strategy diversity contracts, the count of distinct behavioural clusters falls while within-cluster similarity rises. That narrowing is not a failure on its own: the paper’s v1 and v13 strategy maps for all three lineages show what section 7.1 calls a preference for subtractive improvement, and section 7.2 treats the convergence of the sliding-window lineage (Miri) as intended, while noting that Katalin’s selection rule keeps a mix of historical modes in its training pool. It becomes a warning sign when narrowing comes with falling large-k coverage or falling task performance.

A gradual contraction of large-k coverage and strategy entropy is the usual signature of diversity loss. A trajectory where a lineage posts rapid early gains, oscillates for several generations and then drops abruptly over one or two iterations is a different pattern, and the two should not be assumed to share a cause. The paper’s text reports no large-k pass@k or diversity series for that lineage’s outputs (its Figure 14 plots basin-dynamics and transition entropies, discussed below), so the table below sets out how we currently read it rather than a measured diagnosis.

Training data selection rules worth testing for multi-generation updates

In our own experimentation we found that training data selection across generations needs to maintain behavioural continuity rather than pulling isolated score peaks from distant iterations. One option is a recency window, as in sliding-window SFT curation, which restricts candidate episodes to the most recent generations (such as the last three iterations) so the training data reflects environment states close to the current policy.

Within a recent window, replacing a hard top-k cutoff with stochastic prioritisation would keep some lower-scoring trajectories alongside peak episodes. This heuristic adapts the experience replay sampling method from Schaul et al. in Prioritized Experience Replay, which replays important transitions more often. This is an approximate adaptation: the TD-error priorities of value-based RL do not transfer directly to SFT, where episode scores measure environmental effect.

To stop a single outlier iteration from dominating the sampled pool inside the window, one heuristic worth testing is to pair softened score sampling with a per-generation cap, optionally with per-task coverage quotas in the spirit of coreset selection or data pruning. In the snippet below, the cap limits each generation to a set fraction of the target quota k, which reduces the chance that one round dominates the update.

import numpy as np

def select_rows(episodes, window=3, k=5000, alpha=0.6, cap=0.5, seed=0):
    """episodes: dicts with 'generation', 'success', 'score' (>= 0), 'rows'.
    Draws from the last `window` generations only, samples successes by
    softened score instead of hard top-k, caps any generation at `cap`."""
    rng = np.random.default_rng(seed)
    latest = max(e["generation"] for e in episodes)
    pool = [e for e in episodes
            if e["success"] and e["generation"] > latest - window]
    pri = (np.array([e["score"] for e in pool]) + 1e-3) ** alpha
    order = rng.choice(len(pool), size=len(pool), replace=False, p=pri / pri.sum())
    chosen, per_gen = [], {}
    for i in order:
        g = pool[i]["generation"]
        if per_gen.get(g, 0) >= cap * k:
            continue
        chosen.append(pool[i])
        per_gen[g] = per_gen.get(g, 0) + 1
        if len(chosen) >= k:
            break
    return chosen

The selector counts episodes, so k and cap refer to episodes rather than rows. After flattening the chosen episodes into training rows and appending a small rehearsal set, you can pass them to SFTTrainer in the Hugging Face TRL library to fit a fresh LoRA adapter on the unaltered base model. In the paper all three lineages used that same setup, a fresh LoRA on the base model plus 500 rehearsal rows, so it does not isolate how much each part contributes.

When selecting peak trajectories is still the right engineering choice

Our single-run sandbox result does not imply that peak-trajectory selection fails everywhere; we hypothesise that the right curation strategy may depend on whether the agent’s actions change the environment it learns from. When the task distribution is stationary and a reliable verifier exists, such as in formal mathematics, competition programming, or unit-tested code synthesis, best-of-N sampling, rejection-sampling fine-tuning, and RLVR with GRPO work reliably. Where an accurate automated checker is available on a stationary problem distribution, RLVR with GRPO is often the stronger default because the policy gradient sees the failures as well as the successes, although Yue et al. report that RLVR can narrow large-k coverage relative to the base model.

Our recency-windowed SFT workflow is an alternative we are investigating for stateful, shifting environments; in the paper’s setting plain SFT on successful trajectories was more stable than PPO, GRPO or DPO, and we do not present it as an established replacement for RLVR.

FAQs

What is mode collapse in LLMs?

Mode collapse is a reduction in output and strategy diversity where a language model concentrates probability mass onto a narrow set of phrasing templates or reasoning paths. Single-sample accuracy often looks normal, but repeated sampling produces nearly identical responses and fails to cover valid alternative solutions across the task space.

How does diversity loss differ from model collapse?

Diversity loss narrows a model onto a few high-scoring behaviours while preserving fluency, whereas model collapse degrades overall output quality across recursive generations. Model collapse happens when synthetic outputs replace real training data rather than accumulating alongside it, progressively erasing tail knowledge and causing grammatical or factual degeneration.

How do you detect diversity loss in an LLM?

You can detect diversity loss by comparing pass@k at small k against pass@k at large k across training checkpoints and by measuring semantic or action dispersion across sampled outputs. When pass@1 stays flat or improves while large-k coverage drops towards the pass@1 value, the model has lost alternative solution paths.

Is a fast score climb followed by a sharp drop always diversity loss?

No, a rapid score climb followed by a sudden drop is not always caused by diversity loss. In our single sandbox experiment, the peak-selecting lineage climbed to v5, oscillated until v11, then fell sharply; the paper’s analysis describes its training data as a mixture of strategies from different historical modes, which we tentatively interpret as gradient conflict.

Does training on the best episodes cause diversity loss?

Not necessarily, and we have not shown that it does. In our one experiment, the paper’s formal analysis says the peak-selected training set mixes strategies from different historical modes and attributes the collapse to an incoherent training signal, although its figures show fewer late strategies and falling transition entropies for that lineage.

How should you select training data to avoid collapse of either kind?

For recursive model collapse, Gerstgrasser et al. found that accumulating data alongside real data avoids it. In our single sandbox experiment, both full accumulation and a three-iteration window avoided the late score drop, while softened score sampling and per-generation caps are untested heuristics we suggest trying.