TL;DR
- LLM-as-a-judge uses one model to score another model’s output against a rubric. It is the only realistic way to grade thousands of answers instead of a few dozen.
- The method is only as trustworthy as three things behind it: the rubric design, the bias controls, and a validation run on answers with known scores.
- Scoring one answer and comparing two answers are different jobs, and using the wrong one is a common mistake.
Key Takeaways
- Single and pairwise judging answer different questions. A single score tells you how good an answer is; a pairwise verdict only tells you which of two wins, and can’t become a leaderboard score on its own.
- Rubric design decides what the judge can detect. A vague rubric lets a confident, wordy answer score well. A gate, one requirement that zeroes the score if missed, closes that gap.
- Position, self-preference and verbosity bias are well documented. Judges favour the answer shown first, output from their own model family, and longer answers, unless you control for it.
- Validate the judge on fixtures before it grades real output. Answers with known scores catch double-counted credit that a normal-looking judge response would hide.
Grading a model’s answer used to mean a person reading it. That works for a few dozen answers, not for the thousands a benchmark produces, and not at the speed teams now ship model updates. LLM-as-a-judge is the answer most of the field has converged on: have one model score another’s output, at a fraction of the cost and time of a human panel.
What is LLM-as-a-judge?
LLM-as-a-judge is the practice of using a language model to score or compare another model’s output against a rubric or reference answer, in place of a human reviewer. It is faster and cheaper than a human panel, but it brings its own failure modes, from position bias to inventing a score.
The appeal is straightforward. No team can put a few hundred benchmark questions in front of human graders for every model release. A judge model can. The rest of this guide is about what has to be true of that judge before its score is worth trusting.
We run one ourselves. Groundtruth, our dynamic benchmark for geological reasoning, grades every answer with openai/gpt-5.5 as judge. Most of what follows comes from what broke while running that judge in production, and what we built to catch it.
Single vs. pairwise judging
A judge can answer two different questions, and they require different setups: the same split documented in Google’s agent evaluation metrics templates, which we used as a reference when designing our own.
Single (absolute) scoring
Pointwise scoring shows the judge one answer, the rubric and usually a reference answer, and asks for a score against fixed components. That is one call per answer graded. This mode produces a number you can track over time, compare across a leaderboard, or set a pass/fail bar against. It’s also the mode the gate structure described below applies to directly: the judge either finds the core claim in this one answer or it doesn’t.
Pairwise comparison
Pairwise puts two answers in front of the judge and asks which is better. It returns a verdict like “A is better” or “A is much better,” rather than a 0 to 10 score. Done properly, pairwise LLM evaluation costs two judge calls per question: the same pair graded once in each order, so the answer shown first isn’t quietly favoured. That second call exists purely to catch position bias, one of the best-documented failure modes in LLM as a judge evaluation.
Pairwise comparison is the right tool when the question is “which of these two is better,” and the wrong one when the question is “how good is this, on its own.” A pairwise verdict alone can’t become a leaderboard column, because it never produces an absolute score to aggregate.
Single vs pairwise at a glance
| Single (pointwise) scoring | Pairwise comparison | |
|---|---|---|
| What the judge sees | One answer, the rubric, usually a reference answer | Two answers to the same question |
| What it returns | A score against fixed components, such as 0 to 10 | A verdict: “A is better”, “A is much better” |
| Judge calls per question | One per answer | Two, one in each answer order |
| Answers the question | “How good is this answer?” | “Which of these two is better?” |
| Can feed a leaderboard | Yes, scores aggregate across questions | Not on its own; there is no absolute score to add up |
| Main bias to control | Verbosity, handled by a gate in the rubric | Position, handled by swapping the order |
In practice the two work together. We score every answer pointwise for the leaderboard, and use pairwise comparison when the question is whether a fine-tuned model beat its own base model.
LLM-as-a-judge rubric design and calibration
The rubric a judge grades against decides what the judge can find, before the judge ever runs. A rubric that rewards a fluent, comprehensive-sounding answer will pass one, whether or not it commits to the claim being tested.
A gate stops fluent answers collecting partial credit
The fix that does the most work in our dynamic benchmarking rubrics is a gate. One component tests the single claim or mechanism the question exists to check, and missing it zeroes the entire answer, however good the rest is. Without a gate, a long answer that covers everything adjacent to the question collects partial credit everywhere. It lands at a respectable score while missing the point entirely.
Calibration fixtures catch double-counted credit
A rubric that looks correct on a read-through can still double-count credit, and rereading it won’t show the defect. Running fabricated answers through the actual judge will. Write one per point on the scale, attach a predicted score (a calibration fixture), and compare what comes back. Across two 50-question benchmarks we built this way, 14 and 16 fixtures out of 200 scored higher than their author predicted. The cause was the same both times: a component awarded credit for something the gate already required. Every answer that passed the gate collected that component automatically, whether it showed the thing being scored or not. How to Benchmark AI Models puts these numbers in the wider construction process.
Calibration doesn’t run itself, either. In our own harness, fixture checking is a deliberate manual pass that the scoring run never performs on its own. You have to build it into the process on purpose. AI evaluation rubric design that skips this step is passing a rubric no one has actually tested.
LLM-as-a-judge bias: position, self-preference, and verbosity
The foundational LLM-as-a-judge paper, Zheng et al.’s “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, found that a strong judge like GPT-4 can reach over 80% agreement with human preferences. That matches the agreement rate between two human graders. The same paper also named and measured position bias, verbosity bias and self-enhancement bias as the recurring failure modes behind the gap that remains.
Position bias
Position bias is the tendency to favour whichever answer appears first, independent of its content. The direct defence is to judge the same pair twice, once in each order, and treat a flipped verdict as no preference. That is why pairwise comparison costs two calls, not one.
Self-preference bias
Self-preference bias shows up when a judge favours output from its own model family. It is a real conflict of interest whenever a judge and a leaderboard candidate share a lineage, such as a GPT-family judge scoring GPT-family candidates. The honest response is to disclose it in the methodology rather than leave readers to notice it themselves.
Verbosity bias
Verbosity bias rewards length and confidence over correctness. A gate structure closes off most of it on its own. Components score specific things decided in advance, not an overall impression of thoroughness, so a longer answer that never commits to the tested claim gets nothing extra for its length.
LLM-as-a-judge best practices: validating a judge before deployment
None of the structural fixes above catch every way a judge can go wrong, because some failures aren’t about the judge’s reasoning at all. They’re about what it was actually asked to grade.
When the judge grades the wrong question
In one benchmark we ran, the harness pulled the question text for the judge from the answer file rather than from the rubric. When an answer file carried an empty or mismatched question field, the judge graded against the wrong question, or an empty one. An external format conversion can introduce exactly that kind of change quietly. The harness’s own logging only printed a warning and kept running. The output looked perfectly valid, because the judge answered the question it was actually given. A contract validator that only checks output shape has no way to catch it.
Two checks, not one
That’s the practical case for two separate checks. A structured output contract catches malformed or self-contradictory responses; ours has eleven rules a response has to pass. The judge must return machine-checkable JSON with a gate decision, each component’s award and reasoning, and a flag for anything it can’t confidently resolve. Ours rejects and re-grades up to three times, then records the question as unscored rather than accepting a guess.
Fixture calibration, run before any real grading, catches a rubric that’s structurally wrong. Neither check catches a judge given the wrong input entirely. That’s why spot-checking real transcripts, not just the judge’s output, stays a manual step even after both automated checks pass.
LLM-as-a-judge vs human evaluation
On agreement rate alone, a strong LLM judge can match human-to-human agreement on the same task. It doesn’t match a human’s ability to notice that a judge was handed the wrong question entirely. That class of error still needs a person spot-checking real transcripts, not just judge output.
FAQs
What does LLM-as-a-judge mean?
LLM-as-a-judge is the use of a language model to score or compare AI-generated outputs against a rubric or reference answer, replacing a human reviewer for evaluation at a scale humans can’t sustain: thousands of answers, re-run every time a model changes.
Is LLM-as-a-judge reliable?
It can be, conditionally. The founding research found strong judges reaching over 80% agreement with human graders, matching human-to-human agreement, but that number holds only when position bias, self-preference bias and verbosity are controlled for, and the rubric has been calibrated against known answers first.
What is the difference between single and pairwise judging?
Single (pointwise) scoring grades one answer against a rubric and produces a number you can track or rank. Pairwise comparison grades two answers against each other and only says which is better, not by how much: useful for head-to-head questions, useless as a leaderboard score on its own.
How do you reduce bias in an LLM judge?
Run every pairwise comparison in both answer orders to cancel out position bias. Disclose any shared lineage between judge and candidate models rather than ignoring the conflict, the way we do for our own judge in AI Agent Evaluation: Frameworks, Metrics, Benchmarks, and Best Practices. Use a gate-based rubric so length and confidence can’t substitute for the specific claim being tested.
