TL;DR: Measured in an embedding space of behaviors, the strategy diversity of a self-training agent usually falls as its performance rises. That is not a defect: training on filtered successes mostly subtracts strategies that never worked, and the ones that remain deepen into a few basins of attraction that the environment keeps rewarding. The job is to tell that pruning apart from collapse, which needs the diversity curve logged next to success rate, large-k pass@k and a general held-out benchmark.
What Strategy Diversity Measures
Tracking how an agent’s behavior changes from one generation to the next is a core part of recursive self-improvement.
Strategy diversity is the spread of an agent’s behaviors across one generation of episodes, measured at the level of the strategy rather than the tokens. Two episodes that solve a task by reading the error, patching one line and re-running are the same strategy even when their token sequences share almost nothing. The measurement therefore starts with behavioral embeddings: a short summary of each successful episode (the plan, the tool calls, the order of checks) goes through a text embedding model, such as an open-weight Qwen3-Embedding checkpoint, and the vectors are projected with PCA fitted once on an early generation so that later generations stay comparable.
Two numbers come out of that space per generation. The first is the mean squared distance of each episode’s vector from the generation’s centroid, the variance of the behavior cloud. The second is the number of clusters a density-based method such as HDBSCAN finds, which is the number of distinct strategies in use. Behavioral clustering gives you the count; the variance tells you how far apart those strategies sit.
import numpy as np
from sklearn.cluster import HDBSCAN
from sklearn.decomposition import PCA
def strategy_diversity(embeddings, pca=None, n_components=32):
"""embeddings: one vector per successful episode, computed from a
summary of the episode's plan and tool calls, not its raw tokens."""
if pca is None: # fit once, on an early generation
pca = PCA(n_components=n_components).fit(embeddings)
z = pca.transform(embeddings) # reuse the same pca every generation
centroid = z.mean(axis=0)
variance = np.mean(np.sum((z - centroid) ** 2, axis=1))
labels = HDBSCAN(min_cluster_size=5).fit_predict(z)
n_basins = len(set(labels)) - (1 if -1 in labels else 0)
return variance, n_basins, pcaAdditive Intuition, Subtractive Reality
Why Strategy Diversity Falls as Success Rises
The default picture of continual learning is additive: each generation invents a capability the previous one lacked, and the repertoire grows. For an agent that learns from its own filtered successes, that picture is mostly wrong. Yue et al. (2025) showed for RLVR that the reasoning paths a trained model produces were already inside the base model’s sampling distribution, and that training raises pass@1 while narrowing the set of problems solvable at large k (RLVR reasoning boundary study). The training signal selects among strategies the model could already produce. It rarely creates a new one.
Strategy Pruning in Filtered SFT
That is not a criticism of RL. Where a reliable verifier exists, RLVR remains the stronger default, and the sharpening is the point. The same dynamic also appears when the training step is supervised fine-tuning on the episodes that succeeded: every strategy in the base model’s repertoire gets a verdict from the environment, the ones that fail stop appearing in the training set and lose probability mass, and the ones that succeed gain it.
The mechanism is strategy pruning, and it explains why strategy diversity and performance move in opposite directions. Removing the strategies that never worked is worth more, per generation, than any single new strategy could be. Under a single environmental reward, no method that gets the reward is a bad method, and a loop that keeps only such methods is doing exactly what it was built to do.
Eigenform measured this directly in Survival is the Only Reward. Over thirteen versions of three agent lineages trained on their own successful episodes, PCA cluster maps of every strategy generated show the first version spread across many approaches and the last version concentrated in a few tight clusters. Strategies were occasionally added, but far more were amalgamated, refined, compressed or abandoned, and new basins of attraction appeared at a substantially lower rate than existing ones were pruned. The pruning was not general forgetting: HumanEval pass@1 stayed within a few points of the base model across all three lineages. The paper’s name for the mechanism is negative-space learning, a map of what works built by elimination.
Basins of Attraction in Strategy Space
Treat one generation as a map from the current policy’s distribution over strategies to the next. A strategy that succeeds is written into the training set, becomes more probable, is sampled more often, and succeeds more often. That feedback makes successful strategies attractors: once the policy is near one, each generation pulls it closer. The region of strategy space that drains into a given attractor is its basin, and the generation-to-generation picture is a landscape of basins getting deeper and fewer.
Consolidation, Dissipation and Nucleation
Three motions account for most of what the diversity curve shows. Consolidation merges neighboring basins when their shared core (read the output, check it, retry) is reinforced more than their differences, so two clusters become one with a lower variance. Dissipation empties a basin whose strategy stopped being rewarded, either because the environment moved or because a neighboring basin now solves those tasks first. Nucleation, the rare third motion, starts a new basin from an unexpected success, most often a recombination of two existing strategies rather than an invention.
Quality-diversity optimization does the opposite on purpose. MAP-Elites keeps a grid of behavior cells and stores the best solution found in each one, because its goal is coverage of the behavior space rather than one winner (MAP-Elites paper). A self-training agent under one environmental reward empties cells instead of filling them. Both are correct for their objective, and the contrast is useful: the behavioral repertoire that MAP-Elites preserves is exactly what the agent gives up as it converges.
Strategy convergence is therefore the expected end state, not an anomaly. The policy settles into the few basins the environment pays for, and the variance left in the embedding space reflects real differences between task types rather than noise in how the agent approaches the same task. Self-induced distribution shift is what disturbs that end state: the agent’s own actions move the environment, a basin that was rewarded dissipates, and the landscape has to settle again.
Healthy Pruning or Collapse: Reading the Curve
Falling diversity on its own is uninformative, because pruning and collapse produce the same first symptom. The difference is what else moves with it.
| Signal | Healthy pruning | Collapse |
|---|---|---|
| Diversity variance | Falls, then plateaus above zero | Falls toward zero and keeps falling |
| Cluster count | Stays at or above the number of distinct task types | Drops below it; unrelated tasks share one strategy |
| pass@k at large k | Holds; the lost strategies were the ones that failed | Falls; strategies that solved rare tasks are gone |
| Held-out general benchmark | Holds within noise | Drifts down over generations |
| Policy entropy | Falls slowly and stabilizes | Falls fast and monotonically |
The entropy row has a known mechanism. Cui et al. (2025) showed that under RL a high-probability action with a high advantage lowers policy entropy, that the covariance term driving this stays positive through training, and that downstream performance becomes predictable from entropy alone once collapse sets in (entropy mechanism study). The same force acts on filtered SFT: the most probable successful strategy is the one most often written into the next training set, and each round concentrates mass on it.
Strategy Cluster Count as an Early Warning
The cluster-count row is the one to watch first. A repertoire with one strategy per task type has pruned everything redundant and nothing necessary. A repertoire with fewer clusters than task types has merged basins that should have stayed apart, and the next task that needed the merged-away strategy will fail without a fallback. Variance deletion is the selection-side cause of that merge: a filter that keeps only the top-scoring episodes removes the ordinary successes that kept the basins distinct.
Keeping the Repertoire You Will Need
Log the diversity variance and the cluster count every generation, next to success rate and large-k pass@k, and plot them together. The healthy shape is a strategy diversity curve that falls while success rises and then flattens, with a cluster count that matches the task mix. A curve that keeps falling after success has plateaued is the earliest warning you will get, generations before a held-out benchmark moves.
Keep an archive of strategies even when you do not train on it. The behavior-performance map of Cully et al. (2015) let a damaged robot recover in minutes by trying stored behaviors that had scored well before the damage, because the map held the behavioral repertoire that the single best gait had discarded (robot adaptation paper). For an agent, the equivalent is the best episode per behavior cell, stored outside the training window, so a basin that dissipated can be re-seeded with a few rows when the environment shifts back.
Archiving Strategies With pyribs
That archive is a standard quality-diversity structure, and pyribs implements it with an ask-tell interface over a grid of behavior cells. You do not need its emitters for this use. The archive alone, keyed by the same embedding cells the diversity metric uses, is enough to hold the strategies the training window has let go.
Do not add a diversity bonus to the reward by default. Diversity is a diagnostic, not an objective: the environment pays for outcomes, and a strategy kept alive only by a bonus is one the agent will have to unlearn later. Intervene on the selection rule instead. Widen the training window, sample successes stochastically rather than by top score, and cap the share any one strategy cluster takes of the next training set once the cluster count drops below the task-type count.
FAQs
Is the diversity metric the same as policy entropy?
No. Policy entropy measures the randomness of the next-token distribution and can stay high while every episode follows one plan, or fall while several distinct strategies remain in use. The diversity metric is computed on embeddings of whole-episode strategies, so it counts plans, not tokens. Log both, because they fail in different ways.
Why does pass@k at large k matter for diversity?
Large-k pass@k is the outcome-side view of the same repertoire. If a pruned strategy never solved anything, removing it changes nothing at large k. If it was the only strategy that solved some rare task type, the large-k score drops the moment its basin dissipates, which makes it the cheapest check on whether pruning removed something necessary.
Which text should I embed to get behavioral embeddings?
Embed a structured summary of the episode rather than its raw transcript: the plan, the ordered tool calls and the checks the agent ran. Raw transcripts embed surface features such as file names and error strings, which cluster by task rather than by strategy. Fit the projection once, on an early generation, and reuse it so later scores stay comparable.
Is falling strategy diversity a sign that an agent is failing?
Not by itself. In self-training, selection usually prunes strategies that never worked, so diversity falls while success rises. It signals collapse only when the cluster count drops below the number of task types, large-k pass@k falls, or a held-out general benchmark drifts down alongside it.
How do I stop strategy diversity from collapsing in self-training?
Act on the selection rule rather than adding a diversity bonus to the reward. Widen the training window, sample successes stochastically instead of by top score, and cap the share any one strategy cluster takes of the next training set. Keep an archive of past strategies so a dissipated basin can be re-seeded.
