TL;DR
- Benchmarking an AI model reliably means controlling question construction, scoring methodology and statistical resolution, not just running a model against a fixed question set once.
- The framework has three stages. Construct the benchmark, run it so it isolates what you are measuring, and interpret the result against what the sample size can detect.
- The case study is Groundtruth, our own dynamic benchmarking system, including a rubric defect that a human review missed and a mechanical check caught.
Key Takeaways
- Benchmark construction choices are benchmark results. Question design and gating decide what the benchmark can even measure.
- Statistical resolution determines what gaps are detectable. On ours, the smallest detectable gap ranged from 0.27 to 0.45 points depending on the task set.
- Custom benchmarks need machine-gated authoring to stay consistent. Manual authoring drifts across waves.
- Comparing models requires holding the harness fixed. Otherwise the harness, not the model, explains the gap. We found this running eight models through one shared harness.
Search “how to benchmark AI models” and most guides stop at the same place. Pick some tasks, write a scoring rubric, run your model, read the score. That gets you a number. It doesn’t tell you whether the number means anything.
This is the framework we use for the Groundtruth benchmark, which we built and run for geological reasoning, with results on the leaderboard. It’s written to generalise past geology. The numbers below are ours, and the stages are the same regardless of domain.
What does it mean to benchmark an AI model?
How to benchmark AI models comes down to three separable stages: construct a task set that tests what you think it tests, run it so the model is isolated from everything wrapped around it, and read the score against the resolution your sample size supports. Treating any stage as automatic is where reliability breaks down.
The part most guides skip is the middle and the end. A rubric can look complete and still double-count credit or pass the wrong reasoning, and none of that shows up by reading the rubric. It shows up when you run known answers through it and see what comes back.
Step 1: Constructing the benchmark
Before writing a single question, decide what a good outcome looks like. Describe it in the words of someone who does the work, not as a generic accuracy target. This decision also settles whether you need a custom LLM benchmark at all. A public benchmark is enough when general capability is what you’re measuring, a model’s coding ability, its factual recall, its reasoning on textbook problems. It stops being enough once you need to know how a model performs on your own documents, your own edge cases, or a task shape no public benchmark tests.
A generic geology benchmark score told us nothing about whether a model could reason correctly over one company’s own exploration archive. That gap is why Groundtruth generates a fresh benchmark from a user’s own documents instead of reusing a fixed public one. The same logic is why hardware and inference speed get their own benchmarks, such as MLPerf, rather than a general leaderboard that blurs the thing you actually need to measure.
Tasks that resist the two easy outs
A benchmark task is only useful if a model can’t score well by cheating past the actual question. The first shortcut is answering from memorised general knowledge instead of reasoning over the material in front of it. Obscure, source-specific material forces a model to read rather than recall. The second shortcut is sounding right without committing to a claim. A fluent, well-hedged answer that never states the thing being asked for can still read as good to a lenient grader. Both get closed by grounding every task in specific source material and requiring one concrete claim per question. Our custom LLM benchmark pipeline shows the whole process. It includes the machine checks that catch a lopsided task set before a human ever reviews it.
A rubric that produces a clean signal
Two people, or two runs of the same judge, should reach close to the same score from the same rubric and the same answer. The piece of rubric structure that does the most work here is a gate. One component tests the single claim the question exists to test, and missing it zeroes the answer regardless of the rest. Without a gate, a long, confident answer that covers everything adjacent to the question collects partial credit everywhere. It lands at a respectable score while missing the actual point.
Test the rubric before you test a single model
Write fabricated answers for each question yourself, one per point on the scale, and attach the score you expect each to get. Then run them through your actual judge before it grades a real model. We call these calibration fixtures. Running a full set turned up a defect pattern we wouldn’t have found by rereading the rubrics: across two 50-question benchmarks, 14 and 16 fixtures out of 200 scored higher than their author predicted. The cause was the same each time. A component awarded credit for something the gate already required, so every answer that passed the gate collected that component automatically. On one of those benchmarks, the author’s own careful review found 2 of those cases. The mechanical fixture run found 15 more.
Step 2: Running the benchmark
Once you run agents rather than single model calls, an AI benchmarking framework has to account for more than the model. The tools available, the system prompt and the retrieval layer can change the score as much as the model itself. That is the territory of AI agent evaluation.
How to compare AI models without the harness getting in the way
If two rows in a comparison differ in both model and harness, the gap between them has two possible causes and no way to separate them afterward. Hold the harness fixed and vary only the model, or pin the model and vary only the harness. That is the only way to know which one produced a given result. We built a second leaderboard track specifically to isolate this.
Pick a judge, and make it accountable
If a language model assigns your scores, it’s part of your measurement instrument and needs the same scrutiny as the agent being tested. At minimum, require the judge to return a structured, machine-checkable result rather than free text, and validate it against the rubric before accepting it. Our own validator checks eleven rules before it accepts a score. When the judge can’t produce a valid result after a few attempts, leave the question unscored rather than guessing.
Step 3: Interpreting results
A benchmark score is not exact. To evaluate AI model accuracy against benchmarks and baselines, read every result against a resolution. That is the smallest gap the sample size and grading noise can distinguish from chance, and it depends on how many questions back the score.
On our three task sets, that resolution ranged from 0.27 points on the easiest-to-resolve set up to 0.45 on the hardest, using the same benchmark and the same judge. A 0.3-point improvement in model performance is a real finding on one and pure noise on the other. Model rankings, like our comparison of six models on real exploration archives, are exactly the kind of result this applies to.
Publish that number alongside your results, or calculate it before trusting anyone else’s. A full worked example of the calculation is in a companion piece, Measuring the Resolution of a 50-Question Benchmark.
The other half of reliability is maintenance. A benchmark whose rubrics were shaped by watching a handful of models answer them can end up fitted to those models specifically. Hold some models out of that process, then check they land where their general capability predicts. That is direct evidence the rubric grades unfamiliar models the same way as familiar ones. Re-running a stable reference system periodically catches the same problem from the other direction: it tells you whether a score moved because a model changed or because the benchmark quietly drifted.
Common mistakes in AI model benchmarking
- Skipping rubric calibration. Reading a rubric and trusting it is not the same as running fabricated answers through it. The gap between the two is where most scoring defects hide.
- Comparing models without holding the harness fixed. A score difference with two things varying at once can’t be attributed to either one.
- Trusting a gap smaller than the benchmark’s own resolution. A 0.3-point win means nothing if your sample size can’t reliably detect gaps that small.
- Using an LLM judge with no output contract. An unstructured judge response that can’t be validated is a guess wearing a number.
- Never re-running a stable reference. Without one, a score change can’t be told apart from the benchmark itself drifting.
FAQs
How do you benchmark an AI model?
Build a task set grounded in specific source material, with a scoring rubric that includes a gate. Test that rubric against fabricated answers before it grades a real model. Run the model with the harness held fixed if it’s agentic, and read the result against the benchmark’s own statistical resolution.
What makes a benchmark reliable?
A reliable benchmark has tasks that resist memorisation and vague, confident answers, and a rubric validated with calibration fixtures before real use. It also needs a judge held to a structured, auditable output contract, and a known resolution, so a small score gap isn’t over-trusted as a real difference between models.
How is a custom LLM benchmark built?
You generate questions and rubrics directly from your own source material rather than reusing a fixed public set. Scripts then check the structure, an even difficulty spread and evidence grounding, and a person approves the blueprint once before drafting. The result tests what your model needs to know, not general recall.
What is statistical significance in AI benchmarking?
Whether a score gap is larger than what your sample size and grading noise could produce by chance. On a 50-question benchmark, that resolution floor can run from roughly 0.27 to 0.45 points depending on the task set, so a smaller gap isn’t evidence of anything.
