Benchmark Statistical Significance: Measuring the Resolution of a 50-Question Benchmark

TL;DR

• A leaderboard score printed to two decimals looks precise, but with 50 questions behind it, two close models may not be different at all.
• We measured the minimum detectable gap on our own benchmark: 0.27 to 0.45 points depending on the question set. Across eight models and 1,200 graded answers, only 2 of 7 gaps between neighbouring models were statistically significant.
• The same checks found one question set with its difficulty labels upside down, and showed the grading is fair to AI models it had never seen.

Key Takeaways

• A small score difference isn’t automatically a real one. Below a benchmark’s minimum detectable gap, a difference between two models can be measurement noise.
• Benchmark statistical significance depends on the evidence. A score gap matters only when it clears the noise measured for that question set.
• Confidence intervals turn a ranked list into honest tiers. When two models’ intervals overlap, the data does not support calling one the winner.
• A benchmark should be tested against its own results. Comparing authored difficulty labels with actual scores caught one set where “easy” and “expert” were reversed.
• Grading models the rules never saw is a check for bias. Two strong models graded only after the rules were finished still scored near the top, so the rules reward correct answers, not a familiar style.

Benchmark statistical significance is the question every leaderboard skips: is the gap between two models larger than what chance and grading noise could produce? Groundtruth, our benchmark for geological reasoning, grades models on three districts of 50 questions each, scored 0 to 10 by one fixed AI judge.

The latest results cover eight models and 1,200 graded answers, with scoring code available in the Groundtruth Dynamic Benchmarking repository. This post explains how finely those results can be read, and the checks we ran on the benchmark itself.

What Do LLM Benchmarks Actually Measure?

An LLM benchmark measures performance on one specific task set, under one rubric, graded by one judge, in one fixed evaluation setup. It does not measure general intelligence or universal capability across tasks.

Because benchmarks evaluate narrow conditions, every reported score is an estimate shaped by sample size and grading noise rather than an exact reading of ability. In Groundtruth, for example, models answer three distinct districts of 50 questions each, scored from 0 to 10 by a single fixed AI judge. That design means any reported mean carries statistical uncertainty from both question selection and judge scoring variance. Our separate guide on how to benchmark AI models covers the complete construction pipeline, but interpreting any resulting score requires treating it as a noisy estimate. That inherent uncertainty leads directly to the question of statistical significance.

Why Are Public Benchmarks Not Enough?

Public benchmarks answer general capability questions on standardized tasks, which rarely match the specific documents, rubrics, and workflows of a specialized domain. Standardized evaluations also face distinct methodological challenges. In a systematic study of benchmark saturation across 60 LLM evaluations, Akhtar et al. (PMLR 2026) found that nearly half were saturated, with saturation rates increasing with benchmark age and resilience depending on expert curation rather than public or private access. Separately, research by Xu et al. showed that benchmark leakage into training data can skew model evaluations and comparative rankings.

Furthermore, reporting fine decimal scores without statistical testing obscures whether observed differences are meaningful. In our 50-question Groundtruth evaluation, only 2 of 7 neighbouring-model gaps reached statistical significance, with the rest falling inside measurement noise. A custom LLM benchmark grounds evaluation in domain-specific tasks, but it remains reliable only if it reports its own resolution and confidence intervals. Benchmark statistical significance helps readers tell whether its score gaps exceed the measured noise.

What Does Benchmark Statistical Significance Mean?

A score gap is statistically significant when it is larger than the differences that the sample size and grading noise alone would produce. On a benchmark, that sets a detectable threshold: below it, a gap between two models is not evidence that one is better. This is the practical test behind benchmark statistical significance: compare the observed difference with the uncertainty measured for that question set.

The number depends on how many questions you have, how much scores vary between them, and how consistent the judge is. In other words, it is a property of each question set, so it has to be measured, not assumed.

How Do Confidence Intervals Turn Benchmark Rankings Into Tiers?

Eight Models with 95% Intervals

Each model’s result is its mean score per question with a bootstrap confidence interval at 95%: the range its true mean plausibly sits in, given the questions we sampled. Benchmark confidence intervals show how much uncertainty surrounds each mean; using them shows where scores overlap rather than forcing a false hierarchy. Read together, benchmark confidence intervals make benchmark statistical significance easier to judge because they show whether neighbouring scores are clearly separated or still overlap.

Which Neighbouring Gaps Are Significant?

Testing each pair of neighbours for benchmark statistical significance tells the same story. The gaps between Kimi K3, Claude Sonnet 5, GPT 5.6 Sol and OxAlpha were 0.19, 0.12 and 0.06 points, none of them significant. By contrast, the gap from OxAlpha down to Deepseek V4 Pro was 0.65 and significant. Inside the middle group the gaps were 0.06 and 0.19, again not significant. Finally, the last gap, 0.73 from Gemini 3.1 Pro Preview to GLM 4.7, was significant.

The honest reading is three tiers: four models at the top, three in the middle, and one below. Kimi K3 has the highest mean, yet the benchmark cannot say it is better than Claude Sonnet 5. This is the distinction benchmark statistical significance is meant to preserve: observed order is not proof of a real gap. That is why the Groundtruth leaderboard reports tiers, grouping models that overlap within measurement noise.

What Is the Minimum Detectable Gap in a 50-Question Benchmark?

The minimum detectable gap is the smallest difference in mean scores that a benchmark can statistically distinguish from zero given its sample size and scoring variance. Benchmark resolution, how finely a question set can separate real performance from noise, depends on the question set itself. We measured it separately for each district. In practice, benchmark resolution tells readers which score differences are large enough to interpret as evidence rather than noise.

Resolution Steps per District

For each district we measured three things: the spread between the top and bottom model; the minimum detectable gap, the smallest mean gap distinguishable from zero with 50 questions, measured on a representative model pair; and the number of resolution steps that fit between them, which is the range divided by that minimum gap.

The same gap means different things on different districts. For example, a 0.35-point difference between two models is real on WAMEX and within noise on Technical. Technical can separate about four levels of performance across its range; WAMEX can separate nearly eight. That is why benchmark statistical significance must be evaluated against each district’s measured resolution.

Strong Discriminators and Dead Questions

A strong discriminator is a question where the best and worst model scores differ by at least three points. A dead question, on the other hand, is one where every model scored the same, so it costs a judge call per model and separates nothing. The Technical district has one, and it is the first candidate for replacement.

Why Does a Benchmark Need More Than One Corpus?

The top tier kept its shape on all three districts. However, the lower tiers did not. Deepseek V4 Pro scored lowest of the middle group on Technical (7.34 against 7.62 for both Grok 4.6 and Gemini 3.1 Pro Preview), yet highest of the three on WAMEX (8.46). Grok 4.6 dropped to 7.16 on USGS, below Gemini 3.1 Pro Preview at 7.26.

With one district, any of those orderings would have looked like the answer. Measuring three shows which parts of the ranking depend on the corpus, and which gaps reach benchmark statistical significance only on some of them. The differing results also show why benchmark resolution should be reported per corpus, not assumed to transfer across datasets.

How Do You Check Difficulty Labels Against the Data?

Every question carries a difficulty label set during authoring: Easy, Moderate, Hard or Expert. Averaging all eight models’ scores by label shows whether the labels mean anything.

As expected, Technical declines steadily as the labels get harder. WAMEX’s Expert questions are its hardest, although its Easy questions score below its Moderate ones. USGS runs backwards: models do best on the questions labelled Expert, and the Easy questions separate them most.

Even so, the scores themselves are unaffected, because every question is graded against its own rubric whatever its label. The difficulty breakdown for USGS is not reliable, so we report the inversion and do not quote per-difficulty analysis for that district. An authored label is a prediction, and the results are the test of it.

How Do You Check Grading Rules for Bias With New Models?

Think of a teacher who writes the marking guide while reading five students’ papers. The risk is that the guide ends up rewarding how those five students write, not whether their answers are right. It is the same risk covered from the calibration side in Testing the Rubric Before You Test the Model.

Our grading rules, the rubrics, were written the same way, while reading answers from five AI models. So we did what that teacher should do: we graded two more models whose answers we had never looked at, Claude Sonnet 5 and then OxAlpha. Both are known to be strong models. If the rubrics only rewarded the style of the first five, these two would have scored lower than expected.

They didn’t. Both landed in the top tier on all three districts, a result consistent with the rubrics rewarding correct answers rather than a familiar style.

What Are the Limits of a Single Benchmark Run?

Each model was scored once per district. The judge is not perfectly deterministic even at temperature 0, so a gap close to a district’s threshold could move on a re-run. Our guide to LLM-as-a-judge covers how stochastic judge variance affects evaluation thresholds. Exact positions inside a tier are not definitive claims, so sub-threshold decimal gaps should not decide which AI model to use for geology.

In addition, the judge comes from a model family that competes on the board, a conflict disclosed in full in An LLM Judge That Can’t Invent a Score. GPT 5.6 Sol is in the top tier and places third under that judge. It not winning is some evidence against self-preference, and an independent judge is on the roadmap.

How Do You Build a Custom LLM Benchmark That Can Be Read Reliably?

Understanding how to build a custom LLM benchmark that can be read reliably requires designing for statistical resolution rather than raw point rankings. For the full construction process, see how to benchmark AI models. While authoring workflows can generate a 50-question benchmark from technical literature, structuring an evaluation requires seven practices supported by our findings:

1. Choose question count with the measured threshold in mind. More questions and lower score variance shrink it; across our districts, it ranged from 0.27 to 0.45 points.
2. Use more than one corpus before drawing conclusions about lower-tier models. Top models held their positions across our three districts, but lower-tier models changed rank between corpora.
3. Check which questions separate models, and replace dead questions. Technical had one question where all models scored identically, adding evaluation cost without offering resolution.
4. Treat difficulty labels as predictions and test them against results. Workflows authoring benchmarks from PDF to ground truth dataset can misjudge difficulty, as shown by our inverted USGS labels.
5. Calibrate the rubric, then grade models whose answers were held out during authoring. Claude Sonnet 5 and OxAlpha landed in the top tier on all three districts, a result consistent with the rubrics rewarding correct answers rather than a familiar writing style.
6. Fix the judge, and disclose any conflict with evaluated models. We disclosed that our evaluator shared a model family with GPT 5.6 Sol, which placed third.
7. Report tiers and the measured threshold with results. Confidence intervals group overlapping models into honest tiers instead of overinterpreting insignificant decimal differences. Benchmark confidence intervals and the benchmark’s resolution make those limits visible to readers.

What Are the Rules for Reading Benchmark Statistical Significance?

Interpreting benchmark statistical significance requires distinguishing genuine capability differences from measurement noise. A useful report explains the benchmark’s statistical significance in terms of its sample, judge, confidence intervals, and detectable threshold. Follow these five evidence-backed rules when reading or reporting evaluation scores:

1. Report tiers from confidence intervals, and state which neighbouring gaps are significant.
2. Compare any claimed improvement with the minimum detectable gap of the district it was measured on.
3. Use more than one corpus before drawing a conclusion about the lower half of a ranking.
4. Check authored labels such as difficulty against the scores before quoting a breakdown.
5. Keep some models out of writing the grading rules, then grade them afterwards to check the rules for bias. Report benchmark statistical significance with the observed tiers and resolution, so readers can see which comparisons the evidence supports.

Frequently Asked Questions

What is benchmark statistical significance?

Benchmark statistical significance indicates that a performance difference between two evaluated models is larger than what random chance, question sampling variance, and judge inconsistency could produce. Without statistical testing, small decimal gaps on a leaderboard may simply reflect measurement noise. A statistically significant gap provides empirical evidence that one model reliably outperforms another on that specific benchmark and rubric.

What is the minimum detectable gap?

The minimum detectable gap is the smallest difference in mean scores that a benchmark can statistically distinguish from zero at a chosen confidence level. Any score separation narrower than this threshold cannot be distinguished from zero. In our 50-question evaluations, the minimum detectable gap ranged from 0.27 to 0.45 points, meaning smaller differences between neighbouring models were statistically insignificant.

Why use confidence intervals and tiers instead of ranks?

Point-score rankings create a false impression of precision by treating minor decimal differences as meaningful hierarchy. Confidence intervals show the plausible range where a model’s true performance lies, accounting for sample variance. Grouping models into tiers based on overlapping intervals separates statistically distinct clusters from arbitrary rankings, preventing teams from making procurement decisions based on random scoring noise.

How many questions does a benchmark need?

The required number of questions depends on question count, score variance, and judge consistency. Because these factors differ across domains and rubrics, each question set must measure its own minimum detectable gap rather than assuming a universal threshold. In our evaluation across three 50-question sets, the measured minimum detectable gaps were 0.27, 0.33, and 0.45 points, demonstrating that statistical resolution varies significantly even with identical question counts.

What do LLM benchmarks measure?

To understand what LLM benchmarks measure, recognize that a benchmark evaluates candidate performance on a specific task set, under a designated rubric, scored by a specific judge under fixed test conditions. It does not measure general intelligence. Every benchmark score is an estimate constrained by sample size and scoring noise, providing targeted evidence of performance on the tested corpus.

Are public benchmarks enough for choosing a model?

Public benchmarks evaluate broad tasks that rarely match specialized domain workflows or proprietary data. Moreover, primary research shows standardized evaluations face methodological constraints: Akhtar et al. (PMLR 2026) documented that benchmark saturation affects nearly half of examined tests as benchmarks age, while Xu et al. demonstrated that benchmark leakage can skew model comparisons. Selecting an AI model for production requires evaluating candidates on domain-specific tasks and reporting confidence intervals alongside scores.

How do you build a custom LLM benchmark?

When considering how to build a custom LLM benchmark, teams must curate domain documents, author test questions with strict reference rubrics, and fix an impartial judge. To ensure results can be read reliably, test multiple corpora, eliminate dead questions that fail to separate candidates, and calibrate rubrics against held-out models. Finally, calculate confidence intervals and publish the minimum detectable gap alongside all reported scores.