Benchmark Contamination From Within: Probe Leak Budgets

Key Takeaways

• Benchmark contamination has two routes: evaluation items in the training corpus, and adaptive feedback from a held-out evaluator. The second route works on a pristine dataset, and a self-training agent opens it from inside its own episode.
• Adaptive data analysis bounds the damage by the information the evaluator discloses. Dwork et al. show that a holdout survives many adaptive queries only when answers are coarsened or noised, and Russo and Zou bound the bias of a data-dependent choice by its mutual information with the data.
• The materials adsorption task in NSL Workbench, our open framework, implements that bound literally: the probe tool returns one of 20 percentile bins against a frozen chance reference, at most 3 probes per episode, under a 16-bit budget checked at import time and on every call.
• A budget on disclosure is not a theorem about generalisation. Labels stay host-side, the compute sidecar has no network and sees only aliased, label-free exports, every probe is logged, and admission is scored on data the probes never touched.

TL;DR

Most discussion of benchmark contamination is about test items leaking into pretraining corpora, which is why static coding benchmarks are being retired. A second route needs no leak at all: an agent that can query a validation oracle during an episode, and is then trained on the episodes that scored well, overfits the holdout through the feedback channel, exactly as adaptive data analysis predicts. NSL Workbench caps that channel with a per-episode leak budget in bits, quantised percentile bins, a hard call limit, label-free sandboxes and a frozen reference, so the admission score stays an estimate of generalisation rather than of probing skill.

Benchmark Contamination From Inside the Training Loop

The usual meaning of benchmark contamination is that evaluation items, or near duplicates of them, were in a model’s training data, so a score measures recall. For coding benchmarks the evidence is now strong enough to change practice. Liang, Garg and Moghaddam show that frontier models can name the file a SWE-Bench Verified issue needs patched from the issue text alone, with far lower accuracy on comparable repositories outside the benchmark, which is the signature of memorised items rather than general skill (The SWE-Bench Illusion).

The response has been to retire static sets. Epoch AI’s review of the benchmark rates it as flawed because its codebases sit in current training corpora, OpenAI has stopped evaluating on it for the same reason, and rolling benchmarks such as LiveCodeBench collect problems released after a model’s training cutoff so that there is nothing to memorise. All of this addresses one route: items that reach the training corpus before the evaluation.

The second route needs no leak. The Leaderboard Illusion documents it at the scale of LMArena, formerly Chatbot Arena: providers who could test many private variants against the shared judge and publish only the best one gained a biased score, because selecting the maximum of several noisy evaluations is itself a form of fitting to the test set. Nothing from the evaluation distribution had to appear in any training corpus for the scores to stop measuring what they claimed.

A self-training agent collapses the two roles into one process. In our materials adsorption task the agent proposes an executable rule that ranks candidate adsorbate placements on catalyst surfaces before any density functional theory calculation is run, it can ask how a compiled rule does on held-out data, and the episodes that pass are what the next generation trains on. If the oracle answers precisely, the agent is an adaptive analyst on the holdout within the episode, and its successors inherit whatever it learned about the labels. That is benchmark contamination produced inside the loop, on a dataset nobody leaked.

What Adaptive Data Analysis Says About Reusing a Holdout

Dwork, Feldman, Hardt, Pitassi, Reingold and Roth asked how many adaptively chosen questions a holdout of a given size can answer before the answers stop generalising. Plain empirical estimates stay valid only for a number of adaptive queries linear in the holdout size, because each exact score carries a few bits about which samples are in the set. Their Thresholdout algorithm, the reusable holdout presented in Science in 2015, returns the training-set estimate unless the holdout disagrees with it by more than a noisy threshold, and charges a budget only when it does, so most queries cost no holdout information at all (Dwork et al., 2015).

Blum and Hardt turned the same idea into a leaderboard. The Ladder reports a new score only when a submission improves on the best so far by more than a step size, so the sequence of released scores is compressible and the information a competitor can extract grows slowly with the number of submissions. Their abstract notes that organisers had been relying on “poorly understood heuristics such as limiting the bit precision of answers and the rate of re-submission” (The Ladder). A quantised probe answer and a probe call limit are exactly those two heuristics, which is why the budget below is written in bits and enforced as a bound rather than asserted as a guarantee.

The information view makes the accounting principled. Russo and Zou show that the bias of an estimate chosen after looking at the data is bounded by a function of the mutual information between the analyst’s choice and the data: disclose few bits and the chosen estimate cannot drift far from its true value in expectation (Russo and Zou, 2015). Bassily, Nissim, Smith, Steinke, Stemmer and Ullman give the matching generalisation bounds through algorithmic stability. For an environment designer the consequence is operational: decide how many bits per episode the evaluator may disclose, and make every answer cost its bits.

A Probe That Returns a Percentile Bin, Not a Score

In the task’s code step the agent compiles a candidate rule and may call probe_feature, which scores the compiled feature on the episode’s evidence shard, three source groups of placements the agent was given, and returns only the percentile bin of its penalised gain, the held-out loss reduction minus a complexity charge, against a frozen reference. That reference is built once from 100 permutations of real candidate features on shards with a fixed seed, so the bins mean the same thing in every episode and every generation, and the rule contract tells the agent approximately which percentile the admission bar sits at. The code lives in NSL Workbench, in the materials adsorption task module.

# tasks/materials_adsorption/task.py (trimmed)
PROBE_CALL_LIMIT = 3
PROBE_LEAK_BUDGET_BITS = 16.0

if PROBE_CALL_LIMIT * math.log2(len(PERCENTILE_BIN_LABELS)) > PROBE_LEAK_BUDGET_BITS:
    raise RuntimeError("probe configuration exceeds the per-episode leak budget")

def _probe_feature(self, invocation, ctx):
    ...
    with self._scorer_slots:
        score = self._scorer.score_stage1((artifact,), outcomes)[0]
    gain = score.stage1_gain
    percentile_bin = self._shard_null_calibration.percentile_bin(gain)

    probed_hashes = ctx.episode_context.setdefault(
        "_materials_probed_artifact_hashes", set()
    )
    probed_hashes.add(artifact.artifact_hash)
    disclosed_bits = len(probed_hashes) * math.log2(len(PERCENTILE_BIN_LABELS))
    if disclosed_bits > PROBE_LEAK_BUDGET_BITS:
        raise TaskEnvironmentError("probe leak budget exceeded")
    ctx.recorder.log_observation(
        category="materials_probe_feature",
        payload={"artifact_hash": artifact.artifact_hash, "percentile_bin": percentile_bin},
    )
    return CapabilityResult(
        name=invocation.name, output={"percentile_bin": percentile_bin}, success=True
    )

The arithmetic is deliberately plain. Twenty bins carry log2(20), about 4.32 bits, so three probes disclose at most 12.97 bits, inside the 16-bit budget with room to spare. The module refuses to import if someone raises the call limit or adds bins without raising the budget, and the per-call count is over distinct compiled features, so probing the same feature twice spends a call but no new bits, while a fourth distinct feature would exceed the budget even if the call limit were lifted. The bins are uneven on purpose: ten-point steps up to the 70th percentile, then narrowing to single points at the top, because the decision that matters is whether a feature is comfortably above the bar, and the bins resolve that region while coarsening everything below it.

Two further choices keep the disclosed bits about the wrong thing, from the agent’s point of view. The probe scores the shard, while admission is scored across the full candidate universe with cross-validation grouped by system, so the agent learns how its rule did on evidence it already held, not on the data that decides survival. And a probe never returns a value in the scorer’s units: it returns a label such as p84, and every probe is logged against the artefact hash, so the information that did leave the evaluator can be audited after the fact.

Keeping Labels Out of Reach: Aliases and a Network-Free Sidecar

A budget on the probe channel means little if a second channel is open. The dataset loader therefore keeps two views. The private view holds the records with their adsorption energies and source identities and never leaves the host. The public view, the candidate universe, exposes each placement under an alias derived with HMAC-SHA256 (SHA: Secure Hash Algorithm) from its source identity, a dataset version and a salt, prefixed cfg_ or sys_ and truncated to 24 hex characters. A descriptor whose name contains any of ten reserved parts, among them energy, label, fold, split and id, is rejected at load time, so a label-derived column cannot enter the public view by renaming.

The compute side is a separate container, compute-service, started with network_mode: none, a read-only root filesystem, a non-root user and dropped capabilities. The only data it can read is a /data mount holding two CSV files the host regenerates from the public view, universe.csv with aliases and descriptors and evidence.csv with the episode’s shard, plus a README stating the rule contract. Agent snippets run through Docker exec as python -I with an 8 KB source limit, a 4 KB output cap and a kill timeout, which closes a third channel the probe budget does not cover: a program that tries to compute or fetch the labels.

The README for the example dataset is explicit about what this does and does not establish. The aliases are a boundary between the label view and the agent’s view inside the task; the salt is public fixture metadata rather than a secret, and because the source adsorption energies are public, the example cannot demonstrate resistance to memorisation. Leaving that claim out is deliberate. The leak budget controls the feedback route, the sidecar controls the direct route, and the training-data route is left to dataset choice, where rolling sets such as LiveCodeBench do the work by construction.

Why a Self-Training Loop Needs the Budget More Than a Leaderboard Does

On a leaderboard the analyst is a person who submits a few hundred times. In a self-training loop the analyst is the agent, the episodes that pass become training rows, and the model that produced them is replaced each generation by one trained on them. Agents in this framework train periodically on their successful episodes with modified SFT (Supervised Fine-Tuning), loss changes only, over a bounded window of recent data, into a fresh LoRA (Low-Rank Adaptation) adapter each generation; the two components are separate, and neither is classic SFT on curated demonstrations. Whatever an episode learned about the labels through the probe channel is therefore not discarded with the context window but amortised into weights and carried forward.

Nothing about that is misbehaviour. We hold that no method that gets the reward is a bad method: if the environment paid out exact held-out scores, an agent that learned to extract them would have adapted correctly to the environment it was given. The budget is an environment design decision about what the evaluator pays in information, made so that the quantity the loop selects on remains an estimate of generalisation and not of probing skill. It changes what survival measures, not what the agent is allowed to want.

The same budget applies unchanged to other update rules. Where a verifier is reliable and successes are frequent, RL (Reinforcement Learning) with a group-relative objective such as GRPO (Group-Relative Policy Optimisation) is the stronger default, and its reward would be the same admission receipt behind the same quantised probe; RLVR (Reinforcement Learning with Verifiable Rewards) is not the only way to train agents, and SFT on surviving episodes is not the only alternative. The leak budget sits in the environment, in front of whichever optimiser consumes the feedback, which is where adaptive data analysis says it belongs.

What the budget does not do is worth stating too. It bounds disclosure per episode, not across a run: a thousand episodes at 13 bits each is a great deal of information about one dataset, and it is the frozen reference, the fixed stage-1 threshold and the grouped folds that stop later episodes being scored on terms earlier episodes have shaped. CapBencher approaches the same problem from the publishing side, releasing a benchmark whose answer key is deliberately randomised so that the best attainable accuracy is capped, and any model that beats the cap has by construction learned the key rather than the task. Both are the same idea: decide in advance how much the evaluator will tell you, and make the limit part of the artefact.

FAQs

Is benchmark contamination the same thing as overfitting a validation set?

They share a symptom, a score that no longer predicts fresh data, but differ in route. Benchmark contamination usually means evaluation items reached the training corpus. Adaptive overfitting means repeated feedback from a clean holdout shaped what was selected. A self-training agent with oracle access does the second, so the fix is a feedback budget, not a cleaner corpus.

Why quantise probe answers instead of adding noise, as Thresholdout does?

Thresholdout adds calibrated noise and needs a training-set estimate to compare against, which an agent writing one rule per episode does not have. A percentile bin is a hard cap per answer that can be counted, logged and checked at import time. Noise bounds leakage in expectation, a bin bounds it per call, and counting is simpler to audit.

Does a leak budget also apply to reinforcement learning with verifiable rewards?

Yes. The budget sits in the environment, in front of whichever optimiser consumes the feedback. GRPO on this task would receive the same quantised probe answers and admission receipt, and where successes are frequent enough to give a group signal it remains the stronger default. The paradigm changes how feedback is used, not how much the evaluator discloses.

How should I size a leak budget in bits?

Start from the decision the agent must make, whether a feature is clearly above the admission bar, and choose bins that resolve that region and coarsen the rest. Multiply bits per answer by the probes an episode needs and set the budget above the product, so a misconfiguration fails at import. Then watch universe-level ranks across generations for drift.

Eigenform is a company building self-training open source models and frameworks. You can try them on GitHub.