TL;DR: When an autonomous agent generates candidate rules and the loop trains on the ones that pass, every false positive becomes training data, so the threshold has to hold against adaptive, correlated and non-Gaussian submissions. The materials adsorption task in our framework calibrates its references offline: a within-system permutation null that fixes the stage-1 threshold, a pooled shard null that turns probe feedback into percentile bins, and a cluster bootstrap over systems whose 5th percentile has to clear the complexity charge before a rule enters the archive. All of them are hashed against the dataset, the compiler and the scorer configuration, and the task refuses to start without them.
Why Benchmark Validity Breaks When an Agent Writes the Hypotheses
One way to get an AI to improve itself without human supervision is a process in the style of RLVR, where the model answers problems that are easy to check, such as writing a maths proof that a formal proof checker confirms or rejects. Where a reliable checker like that exists, it is a strong default. In fuzzier domains, such as finding patterns in experimental data, no proof checker can say whether an answer is right. In our work the model instead writes hypotheses about the data it has access to, tests them by writing code, and is then fine-tuned on the results that hold up. That sounds simple and obvious, but a loop like this can spiral out of control in several ways that are not easy to check, and each one threatens benchmark validity: whether a passing score can be trusted. Below are three known problems and the fixes we use.
This is a core problem in recursive self-improvement, where an agent writes the hypotheses being scored.
A benchmark is valid for a decision when the number it produces means what the decision assumes it means. For a research agent that proposes executable rules, the decision is binary: does this rule carry information about held-out outcomes, or did noise happen to fit? Benchmark validity in that setting is a property of the threshold, and it fails in three distinct ways as soon as the hypotheses come from an agent rather than from a person who fixed one in advance.
Benchmark Validity Failure One: Volume of Hypotheses
The first failure is volume. A loop that proposes 200 rules per generation and accepts each at p < 0.05 admits about ten noise rules per generation by construction. In a self-training loop they become training rows, the next generation proposes more rules like them, and selection on noise compounds the way it does with cherry-picked training data drawn from the top of a noisy distribution.
Benchmark Validity Failure Two: Dependent Pairs
The second failure is dependence. The score is computed on pairwise rankings of candidate placements, and every pair inside one catalyst system shares placements, energies and the descriptor values the rule reads. A t-test that counts pairs as independent samples inflates the sample size by up to the pair cap of 256 per system, and per-system deltas in held-out cross-entropy need not be Gaussian with tens of systems rather than thousands.
Benchmark Validity Failure Three: Adaptive Hypotheses
The third failure is adaptivity, and it is the one no correction factor repairs. The agent reads its feedback and proposes again, so each hypothesis is a function of the benchmark’s earlier answers. Dwork et al. (2014) framed the gap precisely: inference theory assumes a collection of hypotheses fixed before the data are gathered, and once hypotheses depend on previous outcomes the ordinary empirical estimates remain trustworthy only for a number of queries linear in the sample size.
None of this is a complaint about the agent. Under environment-mediated selection the environment is the only ground truth, and reward-persistence collapse means no method that gets the reward is a bad method. If the bar can be cleared by noise, clearing it with noise is a correct adaptation, and an agent that learns to do so has learned the benchmark accurately. The design burden sits upstream, in how the bar is set and whether anything the agent does can move it.
A Frozen Permutation Null Instead of a Parametric Threshold
A permutation test asks what the statistic looks like when the effect under test is removed and everything else is kept. In the materials task the agent writes a rule that assigns a number to each candidate placement of an adsorbate on a catalyst surface, with adsorption energies taken from the Open Catalyst 2020 data as the outcome, and the stage-1 statistic is the mean per-system reduction in held-out cross-entropy that the rule’s pairwise differences buy over an intercept-only model, minus a complexity charge per node of the rule’s syntax tree. The effect under test is the link between the rule’s values and the energies, so the null is built by shuffling energies among the placements of each system and nothing else.
That phrase carries the design. Every system keeps its size, its evaluation axis, its chemistry block, its grouped fold assignment and its sampled pairs; the feature columns are untouched; only the mapping from placements to energies is destroyed. In tasks/materials_adsorption/scoring.py, calibrate_stage1_null does this for a list of hand-written control rules, 100 permutations by default, scoring every control on every permuted dataset with the same code path the live loop uses. From nsl-workbench, trimmed to the loop (outcomes have already been grouped by system):
statistics: list[float] = []
for permutation_index in range(permutations):
permuted: list[PlacementOutcome] = []
for system_alias in sorted(by_system):
group = sorted(
by_system[system_alias], key=lambda item: item.candidate_alias
)
energies = [item.energy for item in group]
rng = random.Random(
int.from_bytes(
_stable_digest(seed, permutation_index, system_alias)[:8],
"big",
)
)
rng.shuffle(energies)
permuted.extend(
replace(item, energy=energy)
for item, energy in zip(group, energies, strict=True)
)
statistics.extend(self._stage1_statistics(candidates, permuted))
return PermutationNullCalibration.create(
percentile=percentile,
permutations=permutations,
seed=seed,
statistics=statistics,
scorer_hash=self.config.scorer_hash,
compiler_hash=compiler_hashes.pop(),
universe_hash=universe_hashes.pop(),
)What the Permutation Null Code Guarantees
Three details in those lines do most of the work. The shuffle is seeded per permutation and per system from a stable digest, so the null is reproducible on any machine. The statistic is the production one, complexity charge included, so a control that is penalised for bloat in the live loop is penalised identically under the null. And the pooled statistics go into PermutationNullCalibration.create, which sorts them and takes the nearest-rank 95th percentile as threshold. With N controls and 100 permutations that bar rests on 100 N null gains, and a rule with no information clears it one time in twenty whatever the shape of the distribution, because the bar was read off the null’s own quantiles.
Freezing the Threshold Against Adaptive Proposers
Freezing is what answers adaptivity. The calibration is computed in Phase 0, from controls the team wrote, before any agent submission exists, and sealed with a SHA-256 over its contents together with scorer_hash, compiler_hash and universe_hash. Change the fold seed, the regulariser, the pair cap, the compiler version or the candidate universe and the hash no longer matches, so the task raises instead of reusing a stale bar. with_calibration copies the threshold into the scorer configuration, and task.py refuses to construct the task without a frozen calibration unless allow_uncalibrated_threshold is set for a plumbing smoke test. The bar the first episode faces is the bar the last one faces, and nothing the agent submits is ever fed back into it. That is the whole of benchmark validity against an adaptive proposer: not a better test, but a bar the proposer cannot move.
Bootstrap Lower Bounds Resample Systems, Not Pairs
Stage 2 asks a second question of a rule that has passed stage 1: given the rules already in the archive, does adding this one still reduce held-out loss? The scorer fits a baseline logistic model on the archive’s pairwise features, an augmented model with the candidate’s feature appended, and records the per-system difference in held-out cross-entropy between them. The mean of those deltas is the point estimate of the marginal gain. The gate does not read the mean.
Cluster Bootstrap Over Systems
_bootstrap_lower_bound draws systems with replacement, as many as there are systems, averages their deltas, repeats that 1,000 times, sorts the replicate means and returns the value at the floor of 5 percent of the replicate count; the complexity charge is subtracted afterwards and marginal_pass requires the result to be positive. The resampled unit is the system because that is where independence plausibly lives. Pairs within a system share placements and energies, the grouped folds already hold out whole systems, and resampling pairs would reintroduce the inflation the t-test suffered from. Miller (2024) makes the same argument for language model evaluations, treating questions as draws from an unseen super-population and recommending clustered standard errors when questions arrive in groups that share a source; a cluster bootstrap is the non-parametric form of that correction.
The replicate stream is seeded from bootstrap_seed salted with the candidate’s rule hash, so the same candidate always gets the same bound and two candidates never share a stream. Any chemistry block with at least two systems whose mean delta falls below minus 0.05 nats vetoes admission regardless of the global bound, since an admitted rule becomes part of the baseline every later candidate is judged against.
| Reference | Question | Resampled | Used for | Fixed |
|---|---|---|---|---|
| Permutation null | How large a stage-1 gain do uninformative rules reach? | Energies within each system, 100 times, over control rules | stage1_threshold behind signal_pass | Phase 0 |
| Shard null | Where does a probed gain sit among null gains on one shard? | Energies within each shard’s systems, 100 times, over the enumerated family | The percentile bin a probe returns | Phase 0 |
| Bootstrap lower bound | Is the marginal gain over the archive stable across systems? | Systems with replacement, 1,000 replicates | marginal_pass and per-axis, per-block diagnostics | Per candidate, seeded by its hash |
| Attainability reference | How does a gain rank against every rule in the enumerated family? | Nothing; the family’s real gains on the full evidence | Telemetry only, never shown to the agent | Phase 0 |
Shard Nulls Turn Probe Feedback Into Percentile Bins
An agent needs some feedback before it commits to a submission, and feedback is the channel through which a benchmark loses its validity. The task allows at most three probes per episode; each compiles a candidate feature, scores it with the stage-1 statistic on one evidence shard, a fixed subset of systems, and returns a percentile bin and nothing else. The bins come from ShardNullCalibration: each shard’s systems are permuted 100 times under a fixed seed, every expression in an enumerated candidate family is scored on every permutation, the null gains of all shards are pooled, and nearest-rank quantiles at twenty cut points from p0 to p99 become the bin edges. The same object records bar_percentile, where the frozen threshold falls inside the pooled null, which is how the agent learns the bar approximately rather than exactly, within a leak budget counted in bits.
Attainability Reference for the Frozen Bar
The attainability reference asks the complementary question: can the bar be reached at all? A script enumerates every single descriptor, every pairwise product and ratio, and 500 random expressions from a fixed template set, scores them against the frozen threshold on the full evidence, and stores the gains; at run time the task logs where each probed or submitted gain ranks in that family, for the people reading telemetry, never for the agent.
What to Carry Into Your Own Agent Evaluation
Five decisions behind these thresholds transfer to any loop in which an agent proposes and a statistical verifier disposes.
- Fix the unit of independence before anything else: the system, the document, the task instance, whatever the agent’s proposals are scored within. Group the folds by it and resample it.
- Build the null by permuting labels inside each unit and preserving everything else, then score the permuted data with the production statistic, complexity charges and all, not with a simplified proxy.
- Freeze the threshold before the agent runs, hash it against the scorer configuration, the data and the compiler, and make the task refuse to start without a matching calibration.
- Gate admission on a cluster-bootstrap lower bound and keep a veto for sub-group degradation, so a mean carried by a few units cannot pass on its own.
- Give the agent coarse, budgeted feedback expressed against the null, and route the finer telemetry to humans.
The resampling itself needs no custom code: scipy.stats.permutation_test and scipy.stats.bootstrap in the SciPy library cover both steps, the latter with percentile, basic and BCa intervals. What none of the library routines does is freeze the result. The hashing and the refusal to run uncalibrated are a few dozen lines, and they are the lines that make the number mean the same thing in the first generation and the last.
When Exact Verifiers Make Statistical Gates Unnecessary
Where a verifier is exact, none of this is needed. If unit tests pass or fail, or a proof checks, the checker is the null, and RL with a group-relative objective such as GRPO is the stronger default. The machinery above is for verifiers that are statistical, where the reward is an estimate and an agent paid per passing estimate will find the estimator’s weak points. In our loop the episodes whose rules clear the frozen bar become training data: agents periodically train on those surviving experiences with modified SFT, loss changes only, over a bounded window of recent episodes into a fresh LoRA adapter each generation, and the two components are separate. Learning from what the environment lets persist, rather than optimising a reward model, is what we call approximate policy-free online learning, and the frozen thresholds are what keep that training set from filling with noise.
FAQs
Why not correct the t-tests with Bonferroni or Benjamini-Hochberg?
Both corrections need the family of tests fixed in advance, and an agent’s family is open-ended and adaptive: the next hypothesis depends on the last answer. A frozen permutation null sets one bar from the null’s distribution and never re-estimates anything from agent submissions, so the false-pass rate per uninformative rule stays at one in twenty however many arrive.
Why shuffle energies within each system instead of across the whole dataset?
Because the live statistic only compares placements inside one system. A global shuffle would mix energies across catalysts and adsorbates and produce a null that is easier to beat than the real task, so the threshold would land too low. Within-system shuffles keep each system’s size, energy spread and pair sample, and remove only the link to the rule.
How many permutations and bootstrap replicates are enough?
The permutation null pools 100 permutations over every control rule, so its 95th percentile rests on thousands of null gains. For the bootstrap, 1,000 replicates put the Monte Carlo error of a 5th-percentile estimate near 0.7 percentage points of coverage, and raising the count is cheap because each replicate is a weighted average of per-system deltas already in memory.
Does a positive lower bound mean the rule will hold on new catalysts?
No. It is an internal validity statement: on these systems, with this model class and archive, the marginal gain is unlikely to be an artefact of lucky systems. Whether pairwise ranking measures the chemistry that matters is a construct validity question, the kind Bean et al. (2025) found poorly handled across 445 LLM benchmarks, and no resampling answers it.
What is benchmark validity for agent-written hypotheses?
Benchmark validity means the number a benchmark produces matches what the decision assumes it means. For agent-written hypotheses, the decision is whether a rule carries real information about held-out outcomes. Volume, dependence between pairs and adaptive proposals each break standard thresholds, so the bar must be set so the agent cannot move it.
