TL;DR

  • An AI grader asked to “score this out of 10” will hand back a number that looks reasonable whether or not it actually checked anything properly, and a bare number like that can’t be audited after the fact.
  • Our own AI grader, used in Groundtruth, our benchmark for geological reasoning, is never allowed to just state a score. It has to show its work in a fixed, structured format listing exactly what it checked and why, and that structured answer is checked automatically before it’s accepted.
  • If that automatic check fails, for example the numbers don’t add up, the AI grader tries again, up to three times in total. If it still can’t produce a valid result, the question is left ungraded rather than given a made-up number.

Key Takeaways

  • A bare score can’t be audited. A number alone doesn’t say which requirement an answer met, which it missed, or whether the grader even noticed the requirement existed.
  • The grader has to show a structured breakdown, not just a total. Every part of that breakdown gets checked automatically: does the math add up, does every required item appear, is nothing invented.
  • A broken breakdown gets rejected and re-tried, not patched. The grader gets up to three attempts from scratch; after that, the question is recorded as ungraded rather than filled in with a guess.
  • Automatic checks catch structure, not judgement. A grader can hand back a perfectly well-formed, internally consistent breakdown that still reached the wrong conclusion, which is why every rubric is separately tested against known answers first.

An LLM judge (an AI model used to grade another AI’s answers) asked to “score this answer out of 10” will return a number, and the number will look reasonable whether or not the judge applied the rubric. A score of 8 with no breakdown isn’t a usable AI model evaluation metric: nobody can tell which requirement the answer met, which it missed, or whether the judge noticed the requirement at all.

Groundtruth Dynamic Benchmarking, our own benchmark for geological reasoning, grades every candidate answer with one fixed judge, openai/gpt-5.5, against a rubric with a gate and graded components. Rigorous AI model evaluation means treating that judge’s own output as something to validate, not just trust: this post covers what the judge is asked for, what the harness checks, and what happens when the check fails.

What goes into the judge’s prompt

For each answer, the harness renders that question’s marking block into a pointwise scoring prompt. The judge receives:

  • the gate’s pass condition, followed by concrete examples of answers that fail it;
  • every component with its points, the gate marked as such, and its credit condition;
  • the required concepts, and a separate list of indicative terms that earn no marks on their own;
  • accepted name variants, so a spelling or transliteration difference is not penalised;
  • a do-not-credit list of plausible wrong answers;
  • marking rules drawn from the rubric’s scoring settings, such as positive marking after the gate and whole-number scores only;
  • the question, the reference answer, the underlying principle, and the candidate’s answer with leaked control tokens stripped out.

The gate’s fail examples are there for a specific reason. A pass condition on its own tends to let a plausible adjacent answer through, and concrete failing answers give the judge something to compare against. The reasoning behind splitting a rubric into a gate plus scored components is covered in full in Dynamic Benchmarking: How We Did It.

The prompt then walks the judge through fixed evaluation steps: check the gate first and score zero if it fails, assess each component independently, never award marks for vocabulary alone, withhold credit for the listed distractors, and treat correctly evaluated competing interpretations as sound reasoning instead of hedging.

The output contract

The judge may reason at whatever length it needs, but it has to finish with one fenced JSON block in a fixed shape:

json 
{
  "gate_passed": true,
  "gate_reason": "one sentence on why the gate passed or failed",
  "components": [
    {"id": "C1", "awarded": 3, "maximum": 3, "reason": "what earned or withheld this credit"}
  ],
  "total": 8,
  "adjudication_required": false,
  "adjudication_note": null
}

The harness extracts the last valid JSON block from the response and checks it against the rubric for that question. It rejects the result if:

  • gate_passed is missing or is not a boolean.
  • the components list is missing or empty.
  • a component id does not exist in the rubric, or appears twice.
  • any rubric component is missing from the list.
  • an award or maximum is not a whole number.
  • a component’s maximum differs from the points the rubric gives it.
  • an award falls outside zero to its maximu.
  • the gate failed but the total or any award is not zero.
  • the gate passed but the total does not equal the sum of the awards.

None of these rules is subtle. Each one corresponds to a way a judge can produce a plausible number that the rubric does not support: inventing a component, awarding a mark the rubric caps lower, rounding to a half point, or letting partial credit leak past a failed gate. Why an LLM-as-a-judge needs a structured output contract like this at all, not just a rubric, is covered in full in that guide.

What happens when the check fails

A rejected result is logged with the specific problem, and the judge is asked to grade the answer again from the same prompt, for up to three attempts in total. If the judge still has not produced a result that satisfies the contract, the question is recorded with a null score and a warning.

A null is deliberate. The alternatives are to take the last number the judge offered or to fall back on a default, and both would put a figure into the results that nobody can defend. Recording a question as unscored rather than guessed is a design principle covered more broadly in AI Agent Evaluation: Frameworks, Metrics, Benchmarks, and Best Practices. A null shows up in the summary, excludes the question from the average, and can be re-scored later.

Network problems are handled separately. Each judge request gets up to four HTTP attempts, with increasing waits between them, before the scoring attempt counts as an error. Every successful response is billed, including the ones the validator rejects, and the harness records the cost of each call as it happens.

Validating the rubric before the judge sees it

The same problem exists one step earlier. The harness reads rubric fields defensively, so a malformed rubric would not crash anything. It would quietly hand the judge empty criteria, and a judge grading against empty criteria still produces plausible-looking scores.

So a rubric is validated as soon as it is chosen, before any answer is generated or scored. The schema version must match, every gate needs a pass condition, and every question needs components with non-empty credit conditions. An older or malformed rubric is rejected with a message naming the questions at fault. That schema check is separate from, and earlier than, the calibration-fixture testing described in Testing the Rubric Before You Test the Model: this one catches a rubric that cannot be parsed, that one catches a rubric that parses fine but grades wrong.

The adjudication flag

Some answers take an approach the rubric’s author did not anticipate. The judge is told to set adjudication_required only when an answer may deserve credit that cannot be verified from the material in front of it, and to explain why in adjudication_note. The harness lists flagged questions at the end of every scoring run, and the public leaderboard shows a count of flags per submission. A flag does not change the score. It marks where a person should look.

One judge, recorded, and disclosed

Every run uses the same judge at temperature 0 with medium reasoning effort, and the judge’s id is written into the run record. An AI model evaluator is only as trustworthy as its own consistency: a score is only meaningful next to other scores from the same grader, changing judges partway through a benchmark breaks the comparison, and letting a model grade its own answers invites self-preference.

There are two limits to state plainly. The judge is not perfectly deterministic even at temperature 0, so a re-run can move a close comparison. And the judge belongs to a model family that is also on the leaderboard. GPT 5.6 Sol lands in the top tier under that judge, in third place, as covered in Which AI Model Is Best for Your Geology?. That it does not come first is some reassurance, and an independent judge is on the roadmap.

What validation cannot catch

The contract guarantees that a score is internally consistent and matches the rubric’s structure. It says nothing about whether the gate decision was right. A judge can return perfectly valid JSON that passes an answer it should have failed.

We found exactly that case while calibrating one benchmark. A fabricated answer reached the right conclusion by an argument the question was designed to reject, and the judge scored it 10 out of 10 against an expected 0, with a clean, valid JSON block. Structural validation is necessary. Testing the rubric with fixtures of known score is what catches judgement errors, and that process is described in Testing the Rubric Before You Test the Model.

The prompt templates are in prompts.py and the validation logic is in main.py in the Groundtruth Dynamic Benchmarking repository. The rubric rules that stop a judge rewarding fluent but wrong answers are covered in Dynamic Benchmarking: How We Did It, and this contract is one piece of the wider question of evaluating AI models, covered in a forthcoming AI Model Evaluation guide.