Benchmark Validity When Agents Write the Hypotheses

Thu, 12 Nov 2026 09:00:00 +0800

TL;DR: When an autonomous agent generates candidate rules and the loop trains on the ones that pass, every false positive becomes training data, so the threshold has to hold against adaptive, correlated and non-Gaussian submissions. The materials adsorption task in our framework calibrates its references offline: a within-system permutation null that fixes the stage-1 threshold, a pooled shard null that turns probe feedback into percentile bins, and a cluster bootstrap over systems whose 5th percentile has to clear the complexity charge before a rule enters the archive. All of them are hashed against the dataset, the compiler and the scorer configuration, and the task refuses to start without them.

Permutation Tests on Eigenform Continuous Learning Tool Tips

Benchmark Validity When Agents Write the Hypotheses