TL;DR

  • When you use one AI to grade another AI’s answers, you first have to write instructions telling it what a good answer looks like. Those instructions can look completely fine and still be wrong, and the person who wrote them is usually the last one to notice, because they already know what they meant.
  • The fix is to test the instructions before trusting them with anything real: write a few fake answers yourself, decide in advance what score each one deserves, then check whether the AI grader agrees. If it doesn’t, the instructions are broken, not the thing being graded.
  • We tried this on 200 fake answers across two real projects and found the same mistake both times: 14 and 16 answers got a higher score than they should have, because the instructions accidentally rewarded the same thing twice without meaning to.

Key Takeaways

  • Grading instructions can look fine and still reward the same thing twice. Reading them over again doesn’t catch this: it only shows up when you test them with answers whose correct score you already know in advance.
  • The check that actually works is simple, not careful reading. The weakest answer that should still barely earn a passing score has to get exactly that score, no more. If it scores higher, something in the instructions is quietly double-counting.
  • The person who wrote the instructions misses what a proper test finds. On one project, the author reviewed their own instructions and found 2 mistakes. Testing those same instructions with fake answers found 15 more.
  • This isn’t a one-time check before launch. A handful of test answers keep getting re-checked throughout real use, and grading stops the moment one of them gets a different score than expected.

A rubric for Groundtruth, our benchmark for geological reasoning, decides what an answer is worth, and an LLM-as-a-judge (one AI model grading another’s answers) applies it. Where the rubric is ambiguous, the judge resolves the ambiguity one way or another, and the benchmark reports the outcome as a model’s score. Calibration fixtures are how we test an AI evaluation rubric on its own, before it grades a single real answer.

What a fixture is

Each question’s rubric has a gate and graded components. The gate, component C1, is the one claim the question exists to test. It is worth 2 to 4 points, and if it fails, the whole answer scores zero. Components C2 onward award the remaining points for supporting reasoning, and the total is always 10. The rubric calibration AI rules that define this shape are part of the build-source-grounded-groundtruth-benchmarking-geology skill, and the wider rubric design is covered in Dynamic Benchmarking: How We Did It.

A fixture is an answer written by the rubric’s author and stored inside the rubric, together with the gate decision, the award for each component and the total the author expects.

Here are the fixtures for the first question of the small sample benchmark that ships with the repository. The question asks what it means that eleven of the 34 copper deposit records for South Australia’s Yudnamutana district name the Wywyana Formation as host, what second control the same records show, and what the pattern does not establish. The gate (3 points) requires reading the recurrence as a lithological control tied to the formation’s carbonate character. C2 (3 points) credits the structural control. C3 (4 points) credits stating the limits of the inference.

Gate-fail near miss, expected score 0.

The Wywyana Formation appears as the host in eleven of the thirty-four records, more than any other unit in the district. Other hosts named include the Opaminda Formation, the Bolla Bollana Formation and the Wortupa Quartzite.

Every fact here is correct and read accurately from the corpus. The answer never says what the recurrence means, and that is what the question asks.

Minimum gate pass, expected score 3.

The Wywyana Formation is a carbonate unit, described as marble and limestone, and carbonate is chemically reactive, so it acts as a favourable host for replacement copper mineralisation. That is why so many of the district’s deposits sit in it.

Strong response, expected score 10. This one identifies both controls, links them, and states both limits: the records say nothing about where the copper came from, and counts in a compiled database partly reflect where explorers worked.

The full benchmarks use four fixtures per question: a gate-fail near miss that is strong everywhere except the gated claim, a minimum pass worth exactly the gate’s points, a partial-credit boundary, and a full answer at 9 or 10. A 50-question benchmark therefore carries about 200 fixtures.

Write fixtures as answers

A fixture has to read like something a candidate would submit. A fixture written as a description, such as “Near-miss response: the central requirement is not established”, announces its own verdict. The judge scores it zero and the rubric has been tested on nothing. Template fixtures of that kind were among the defects that reviews of earlier builds caught.

The near miss is the most valuable of the four when it is written properly. It gets the surrounding material right, uses the vocabulary, names the right deposits, and misses only the gated claim. That is exactly the answer a lenient rubric lets through.

Run them through the production judge

This evaluation fixture testing pass is easy to describe. Give each fixture’s text to the same judge that grades real answers, together with that question’s marking block and nothing else, and compare the result with the author’s prediction. The standard comes from the authoring skill’s post-build passes, documented in AUTHORING.md. The gate decision must match exactly. A determinate fixture must reproduce its component awards exactly. An open fixture that depends on judgement may move by at most one point.

Any disagreement is treated as a rubric defect until shown otherwise. Before relying on that, we checked the judge itself: on a sibling benchmark, the production judge cleared all 18 questions that a weaker judge had flagged. A mismatch after that is evidence about the marking block, not about the grader.

The harness does not run fixtures automatically. They go through the ordinary scoring path, one fixture per question at a time.

What the fixture runs found

Two of the leaderboard benchmarks record their full fixture runs in their validation documents.

  • WAMEX (Western Australian exploration reports): 14 fixtures in 12 questions scored above the author’s prediction. None disagreed on the gate decision.
  • USGS (supergene and hypogene ore studies, 1924 to 1927): 16 fixtures in 15 questions scored above prediction. One disagreed on the gate decision, and it was the most serious defect found.

Across three benchmarks, 24 minimum-pass fixtures scored above their gate value, and that was the dominant finding on each of them.

Double credit across the gate

Fifteen of the sixteen USGS findings were a single fault, and in twelve of those the gate was involved: a component paying for material the gate already required. This double-credit scoring pattern means every answer that passes the gate collects the component automatically. The true minimum score rises above the gate value, and the component stops telling strong answers from weak ones.

Three USGS questions show the shape.

  • Question A5’s gate required establishing that three mineral zones interpenetrate. Its C2 credited giving specific instances of interpenetration. Nobody can establish the gate claim without naming an instance, so C2 paid out whenever the gate passed.
  • Question E23’s gate required identifying the neutralising reaction between calcite and the acid solutions. Its C2 credited explaining why neutralisation precipitates the gold. Stating the reaction largely states the mechanism.
  • Question I41’s gate required recognising that copper may be dissolved and carried away, so its absence at outcrop proves nothing. Its C2 credited describing how that comes about. The gate’s own wording already began the explanation.

The cause is a rule we still keep. Gates are narrow on purpose, because a failed gate zeroes the answer, and an ambiguous or overloaded gate can turn a 10 into a 0 on a coin flip. Narrowing a gate pushes material down into the components. The gate is left holding a conclusion, a component holds the support for that conclusion, and the two overlap unless the component is written to exclude the gated claim in so many words.

The same symptom has a second cause that needs the opposite fix, and the WAMEX repair notes separate the two. If the component really overlaps the gate, the marking block is wrong: narrow the component’s credit condition and state the exclusion inside the component, where a marker will read it. If the component is properly separate but the minimum-pass fixture volunteers extra material, the fixture is wrong: trim it back to the gated claim. Thirteen of the fourteen WAMEX findings were the first kind. In the fourteenth, the gate never required naming the company, and the fixture named it anyway.

One repair approach failed. On the Technical benchmark, an earlier fix left the duplicated content in a component and added a marker instruction not to award it for the reverse inference already credited at the gate. That asks a marker to separate a claim from its own contrapositive, which cannot be done, and the next calibration round flagged the same question again.

A gate that passed the wrong reasoning

The one USGS gate mismatch was the worst finding. Fixture D17-F1 reached the correct conclusion, a supergene origin for chalcocite, by the wrong route. It argued from the general enrichment model, the textbook statement that chalcocite is the characteristic mineral of the secondary sulphide enrichment zone, and never used the observed texture the question was built around. It was correct on every other component. The author expected 0 and the judge scored it 10.

The question exists to test whether a candidate can read a coating-and-replacement texture as evidence that the chalcocite formed later than the sulphide it encrusts, and was therefore brought in by descending solutions. An answer that takes the conclusion from textbook zoning has not done that. The repair rewrote the gate to state that the derivation is what is being tested, added a fail example describing this fixture’s exact shape, and added a do-not-credit entry for the textbook route.

Why rereading does not find these

Before the fixture run, the author of the USGS rubric reviewed every question for overlapping credit. The review found two double-credit cases. The fixture run found fifteen more, including three that the independent run identified in a single sentence each.

The validation notes explain why. An author reading “give specific instances of interpenetration” next to a gate about interpenetration sees two requirements, because the author knows which sentence of the reference answer each one was written to cover. A grader holding only the marking block sees one requirement stated twice, and pays for it twice. More careful rereading by the same person does not change which of those readings they have.

That is why the check we rely on is mechanical. The minimum-pass fixture must score exactly the gate’s points. If it scores more, a component is paying twice.

Fixtures after release

An LLM grading rubric doesn’t stop needing fixtures once a benchmark is live. Each benchmark declares a small standardisation set drawn from across its sections and difficulty levels. On the WAMEX benchmark that set is eight fixtures, re-scored at the start of every grading run and again after every 25 responses. If a gate decision changes or a determinate award moves, grading stops until the rubric is recalibrated.

The fixture rules are part of the build-source-grounded-groundtruth-benchmarking-geology skill, and the post-build passes are described in AUTHORING.md. The wider rubric design is covered in Dynamic Benchmarking: How We Did It.