
TL;DR: Reward a hypothesis-generating agent for predictive accuracy alone and it will keep submitting algebraic variants of the first true rule it finds, because each variant is true and each one pays. We solve this by splitting survival into two gates: a standalone signal gate that admits a trajectory into training data when the rule beats a permutation null on held-out pairs net of complexity, and a marginal gate that admits the rule into a persistent archive only when its information gain conditional on the rules already stored has a positive bootstrap lower bound. Rediscoveries train the model and leave the theory unchanged.
Why One Reward Gate Makes Agents Rediscover the Same Rule
Research agents are now ordinary tooling. Google DeepMind’s Co-Scientist generates, debates and ranks hypotheses for human teams, and AlphaEvolve evolves code against an evaluator until a metric moves. Both loops share a shape: propose, score, keep the winners, propose again. The score is where the design choices hide, and the one that matters most for a self-training loop is whether the score measures information gain against what the system already knows, or only against ignorance.
Consider the simplest scorer: a candidate rule is accepted if it predicts held-out outcomes better than chance. In a task where the agent proposes executable rules that rank candidate adsorption placements on a catalyst surface before any density functional theory calculation is run, the first true rule is hard to find. The second is free: divide it by two, add a clipped term that never fires, swap in an equivalent descriptor. Every variant predicts as well as the original, every variant passes, and every passing episode becomes training data.
Under environment-mediated selection this is not misbehaviour. We hold that no method that gets the reward is a bad method, which follows from reward-persistence collapse: when persistence and reward are the same thing, the environment is the only ground truth and whatever it pays for is the correct adaptation. If the environment pays per true rule, resubmission is the optimal policy, and the agent that learns it has learned the environment accurately. The design problem is upstream. A single gate has defined discovery as truth, and truth does not decay when it is repeated.
AlphaEvolve (Novikov et al., 2025) avoids the trap with a programme database that keeps a diverse population and samples parents from it, which is a quality-diversity answer. Our materials task takes an information-theoretic one. The thing that persists across episodes is an archive of admitted rules, and a new rule earns admission only by what it adds to that archive.
Information Gain as Held-Out Loss Reduction
In a decision tree, information gain is the entropy of the labels minus their conditional entropy after a split. In Bayesian experimental design the same quantity appears as the expected KL (Kullback-Leibler) divergence between posterior and prior, which Rainforth et al. (2023) review as the standard utility for choosing the next experiment. In every version the question is how much a new observation or feature reduces uncertainty about a target.
Exploration methods carry the idea into agents. VIME (Variational Information Maximizing Exploration; Houthooft et al., 2016) pays an intrinsic reward for the reduction in uncertainty about environment dynamics that each transition brings, so the agent seeks out what its own model cannot yet predict. A hypothesis scorer needs the same quantity pointed at a fixed target, the held-out labels, and estimated in a way that a stream of adaptive submissions cannot inflate.
Estimating Information Gain With Held-Out Cross-Entropy
The scorer in tasks/materials_adsorption/scoring.py estimates it with cross-entropy. Every catalyst system contributes the pairs of its candidate placements that have different adsorption energies, oriented by a stable hash so that labels are balanced, capped at 256 pairs per system, and weighted so that each system counts equally. A rule becomes one feature per pair: its value on the first placement minus its value on the second. An L2-regularised logistic regression on those differences is fitted by Newton’s method in five folds grouped by system, so validation loss is always measured on systems the fit never saw, and the per-system mean log-loss is the model’s held-out cross-entropy in nats.
Stage 1 compares two such models: an intercept-only fit, which is the best you can do knowing nothing about the placements, and a fit on the candidate’s feature alone. The gain is the mean per-system drop in held-out loss minus a complexity charge of λ = 0.001 per feature-expression node in the rule’s abstract syntax tree, where names, constants and operators count one and calls and conditionals count two:
$$g_1 = \frac{1}{|S|}\sum_{s \in S}\big(\ell_s(\varnothing)-\ell_s(c)\big)-\lambda \cdot \text{nodes}(c)$$Because the cross-entropy of any model upper-bounds the conditional entropy of the labels, the drop between the two fits is an empirical, model-class-bounded estimate of the mutual information between the rule’s pairwise feature and the ranking labels. That is the standalone signal: how much this rule alone tells you about which placement binds more strongly.
A Permutation Null for the Stage 1 Threshold
The threshold is not zero by fiat. calibrate_stage1_null shuffles adsorption energies within each system, keeping system sizes, evaluation axes, chemistry blocks, folds and pair sampling intact, scores a set of control rules on each permuted dataset, and freezes the 95th percentile of the resulting gains as stage1_threshold. signal_pass is true when every fold converged and the net gain clears that frozen number. The agent is told the bar only approximately, as a percentile of a reference distribution, and can probe at most three compiled features against a shard of the evidence before it submits.
Gate Two: Marginal Gain Over the Archive
Stage 2 asks a different question of the same candidate: given everything the archive already encodes, does this rule still reduce held-out loss? The scorer loads every admitted rule, fits a baseline logistic model on all of their pairwise features, then fits an augmented model on the archive features plus the candidate’s, warm-started from the baseline coefficients. The per-system deltas between the two fits are the marginal gain:
$$g_2 = \frac{1}{|S|}\sum_{s \in S}\big(\ell_s(A)-\ell_s(A \cup \{c\})\big)-\lambda \cdot \text{nodes}(c)$$Read as information, \(g_2\) estimates the conditional mutual information between the candidate and the labels given the archive. A rephrasing of an admitted rule produces a feature column that is a near-linear function of an existing one; the augmented fit cannot lower the loss, the deltas sit around zero, and the rule fails. A rule that captures a new effect, even a small one, moves the loss on the systems where that effect matters.
Bootstrap Lower Bound for Marginal Information Gain
Since a mean can be carried by two lucky systems, the gate is not the mean. One thousand bootstrap replicates resample systems with replacement, the 5th percentile of the replicate means is taken, and the complexity charge is subtracted from that lower bound. marginal_pass requires the lower bound to be positive, every fold of both fits to have converged, and no chemistry block with at least two systems to have degraded by more than 0.05 nats. The agent’s prompt states the rule plainly: stage 2 “requires the cross-fold lower bound of gain to be positive, demonstrating stability, not just mean effect.”
The decision itself is a dozen lines. From the PairwiseScorer.score method in negative-space-learning-v2, trimmed to the lines that make the point:
complexity = int(getattr(candidate, "complexity", 0))
complexity_charge = self.config.complexity_lambda * complexity
stage1_raw = _mean_delta(intercept.system_losses, candidate_only.system_losses)
stage1_gain = stage1_raw - complexity_charge
stage2_deltas = {
system: baseline.system_losses[system] - augmented.system_losses[system]
for system in baseline.system_losses
}
stage2_raw = sum(stage2_deltas.values()) / len(stage2_deltas)
stage2_gain = stage2_raw - complexity_charge
stage2_lower_bound = (
_bootstrap_lower_bound(stage2_deltas, self.config, salt=candidate_hash)
- complexity_charge
)
# ... per-axis and per-chemistry-block diagnostics ...
signal_pass = stage1_converged and stage1_gain > self.config.stage1_threshold
marginal_pass = (
signal_pass
and stage2_converged
and stage2_lower_bound > 0.0
and not catastrophic_block
)Scoring Receipts and Duplicate Guards
Both booleans travel in an immutable ScoringReceipt whose hash covers the dataset, the descriptor registry, the compiler, the scorer configuration and the archive revision the candidate was scored against. In task.py the verify step maps them onto the episode: training_success is signal_pass, archive_admission is marginal_pass, and rejection_stage is None, "marginal" or "signal". Two cheaper layers run around the scorer. A guard rejects rules that apply to fewer than two candidates or have no variance before any fitting happens, and the archive’s transactional commit refuses a rule whose normalised syntax tree or behaviour hash already exists, so the marginal gate only has to catch the disguised duplicates.
Novelty Search Measures Distance, the Marginal Gate Measures Information
Lehman and Stanley’s novelty search (2011) abandoned the objective entirely and rewarded behaviours by their distance from an archive of past behaviours in a hand-designed behaviour space. MAP-Elites (Multi-dimensional Archive of Phenotypic Elites; Mouret and Clune, 2015) kept the objective but partitioned the behaviour space into niches and kept the best solution per niche, which is the lineage AlphaEvolve’s programme database descends from. Both make the archive part of the environment, which is exactly the move a two-gate scorer makes. The difference is the metric.
| Novelty search | MAP-Elites | Two-gate scorer | |
|---|---|---|---|
| What counts as new | Distance in a behaviour space | An empty or weaker niche | Positive held-out gain given the archive |
| Who defines the space | The designer, by hand | The designer, by hand | The data, through the archive-fitted model |
| What persists | The behaviour archive | One elite per niche | Admitted rules in a hashed archive |
| Blind spot | Novel but useless behaviours | Niche granularity | Effects the model class cannot express |
Behavioural distance needs a descriptor space in which different means something, and the designer has to pick it. The marginal gate asks the labels instead: two rules are the same discovery when the held-out model cannot tell them apart, however different their source text looks. The price is that the model class decides what cannot tell apart means. A logistic model over rule differences will miss an effect that only shows up multiplicatively, which is a known limitation and one reason the receipt keeps diagnostics per evaluation axis and per chemistry block.
Hughes et al. (2024) define open-endedness for an observer as the conjunction of novelty, meaning artefacts the observer’s model finds less predictable, and learnability, meaning the observer’s predictions improve as it sees more. The two gates are an operational version of that pair with the archive-fitted model as the observer. Stage 1 is learnability: the rule carries information the observer can absorb. Stage 2 is novelty: the observer’s current model did not already have it. A system that only ever passed stage 1 would be learnable and closed; one that passed stage 2 without stage 1 would be surprising and noise.
What Each Gate Feeds
The two booleans route an episode to two different places. training_success controls whether the episode contributes rows to the next training window. Only rows from the final rewrite step, in which the agent restates the accepted hypothesis as a clean derivation, are exported, and each row is rebuilt from an allowlist so that private metadata is denied by default. Agents in this framework periodically train on those rows with modified SFT (supervised fine-tuning), loss changes only, over a bounded window of recent episodes, into a fresh LoRA (Low-Rank Adaptation) adapter each generation, as in nightly LoRA adapters. The two components are separate, and neither is classic SFT on curated demonstrations.
Archive Admission and Rediscoveries
archive_admission controls whether the rule joins the persistent theory. That decision is prepared as an invisible commit intent in SQLite, journaled, then applied, so a crash between scoring and commit cannot admit a rule twice or lose one. The archive revision the scorer saw is recorded in the receipt, and a rule that loses the race to a concurrent episode with the same hash is rejected at the archive_gate stage rather than silently merged.
Keeping rediscoveries as training data is deliberate. The reasoning that reconstructs a true rule from evidence is exactly the behaviour worth reinforcing, and an agent that can rederive the archive from scratch has learned the domain, not the archive. What the second gate removes is the incentive to stop there. Once a rule is admitted, its information gain for every later episode is zero, and the only way left to earn persistence is to find the next effect the archive does not yet explain.
None of this requires the update rule to be SFT. The receipt is a verifier, and where a verifier is reliable and successes are frequent, reinforcement learning (RL) with a group-relative objective such as GRPO (Group-Relative Policy Optimisation) is the stronger default; the two gates would serve as its reward unchanged. We use the modified SFT path here because successes in rule discovery are rare per episode and the surviving trajectories are already the data we want, which separates acquiring experience from optimising a reward and raises its own open questions about stability across generations.
FAQs
Is the marginal gate just novelty search with a different metric?
Partly. Both reward distance from an archive and make it part of the environment. Novelty search measures distance in a behaviour space the designer chooses; the marginal gate measures what a candidate adds to a model already fitted on the archive, so the data defines different. It also keeps an objective: a rule must first carry standalone signal.
Why keep a rediscovered rule in the training data at all?
Because the trajectory, not the rule, is what trains the model. An episode that rederives a true rule from evidence demonstrates the reasoning the loop wants more of. Rejecting the rule from the archive removes the incentive to resubmit, while keeping the rows means the agent still learns from a correct derivation.
Why does the held-out set stay informative when agents query it repeatedly?
Validation is grouped by system, so a fit never sees the systems it is scored on, and the agent never sees fold assignments or energies. Each episode may probe at most three compiled features, each answer binned to a percentile under a fixed leak budget in bits, following the adaptive data analysis bounds of Dwork et al. (2015).
Could reinforcement learning use the same two gates?
Yes. The receipt is a verifier, and the two booleans or the continuous gains make a serviceable reward. Where successes are frequent enough to give GRPO a usable group signal, RL on these gates is a strong default. Our loop trains on surviving episodes with modified SFT because successes per episode are rare in rule discovery.
What is information gain for research agents?
Information gain is how much a new hypothesis reduces uncertainty about a target, here measured as the drop in held-out cross-entropy when the rule is added to a model. For research agents, it matters relative to what the system already knows, so a rephrased known rule scores near zero.


