TL;DR
- When an AI grades two answers side by side and picks a winner, it tends to favour whichever one it happens to read first, regardless of which one is actually better. A single head-to-head verdict mixes answer quality with plain answer order.
- The fix: ask the same AI grader twice, swapping which answer goes first the second time, then convert both verdicts to one number and average them. An AI that just picks whatever’s shown first ends up with both answers tied, instead of one wrongly winning.
- We use this method in Groundtruth, our own benchmark for geological reasoning, for questions like “did this new version actually improve on the old one.” It doesn’t produce a single score you could rank on a leaderboard; that comes from a separate, simpler method that grades each answer on its own.
Key Takeaways
- A head-to-head verdict alone can’t separate quality from order. Swapping which answer is shown first, then averaging both verdicts, is the only way to know how much of a “winner” came from content rather than position.
- A judge that always picks whatever it sees first produces a tie, not a wrong winner. Averaging a run where it prefers first position with a swapped run where it also prefers first position cancels the effect out cleanly.
- This method answers “did this change help,” not “how good is this.” A head-to-head verdict is relative and can’t become a 0 to 10 score, so it can’t build a leaderboard column on its own.
- An unparseable verdict is recorded as unknown, not guessed. If either the normal or swapped run fails to produce a readable result, both responses are kept for inspection instead of trusting whichever one worked.
Pairwise LLM evaluation gives an LLM-as-a-judge, one AI model grading another’s answers, two answers to the same prompt and asks which one is better. It is a natural fit for comparing a fine-tuned model with the base model it started from, and it is how Groundtruth Dynamic Benchmarking, our own benchmark for geological reasoning, was first used.
It carries a known weakness: LLM judge position bias. Given two answers of similar quality, a judge leans towards the one presented in a particular position, usually the first. If you ask once, you cannot tell how much of a verdict came from the answers and how much came from their order.
Ask twice, swap the order
For every question, the harness sends the judge the same prompt twice. It’s the same swapped-order approach documented in Google’s agent evaluation metrics templates, which we used as a reference when designing our own.
- Normal run. The baseline answer (File A) is shown as Assistant A, and the candidate answer (File B) as Assistant B.
- Swapped run. The candidate answer is shown as Assistant A, and the baseline as Assistant B.
In both runs the judge writes a short evaluation and ends with one of five tokens: [[A>>B]], [[A>B]], [[A=B]], [[B>A]] or [[B>>A]].
The two verdicts cannot be averaged as written, because “A” means a different file in each run. So each is converted to a number from the point of view of File A:
| Verdict token | Value in normal run | Value in swapped run |
|---|---|---|
A>>B | +2 | -2 |
A>B | +1 | -1 |
A=B | 0 | 0 |
B>A | -1 | +1 |
B>>A | -2 | +2 |
The final score is the mean of the two values, and it maps back to a verdict:
| Mean | Final verdict |
|---|---|
| 1.5 or more | File A much better |
| 0.5 to under 1.5 | File A better |
| above -0.5 and below 0.5 | Tie |
| above -1.5 and up to -0.5 | File B better |
| -1.5 or less | File B much better |
Worked examples
| Normal run | Swapped run | Values | Mean | Final verdict |
|---|---|---|---|---|
A>B | B>A | +1, +1 | 1.0 | File A better |
A>>B | B>>A | +2, +2 | 2.0 | File A much better |
A>B | A>B | +1, -1 | 0.0 | Tie |
A>>B | A>B | +2, -1 | 0.5 | File A better |
B>A | A>>B | -1, -2 | -1.5 | File B much better |
The first two rows are consistent judgements: the judge preferred the same file whichever position it appeared in, so the verdict stands, once moderately and once strongly. The third row is position bias LLM judge behaviour in its purest form: the judge preferred whatever it saw first, both times, and the two preferences cancel into a tie. The fourth row is a mixed case: the judge favoured File A strongly in one order and weakly preferred File B in the other, and the average keeps a weak preference for File A. The fifth row is another consistent judgement, this time for File B: a moderate preference in the normal run and a strong one in the swapped run average out to a clear win for File B.
If either run fails to produce a readable verdict token, the harness records the verdict as unknown and keeps both judge responses for inspection, instead of trusting the run that did parse.
What it costs
Two judge calls per question instead of one. Groundtruth’s scoring mode, set as a flag in the Groundtruth Dynamic Benchmarking repository, lets you choose what to pay for when comparing two answer files: the pairwise verdict alone at two calls per question, the pointwise 0 to 10 score for both files at two calls, or both together at four.
What a pairwise verdict cannot tell you
A verdict is relative. “File B much better” on a question says nothing about whether either answer was any good, and swapping in a different baseline can change every verdict. You cannot rank more than two models from pairwise LLM comparison without running every pair, and you cannot turn them into a 0 to 10 score. That is why the Groundtruth leaderboard is built only from pointwise scores, where each answer is graded against its rubric on its own.
The two modes can also disagree for a structural reason. Groundtruth’s pairwise prompt asks the judge to compare both answers against the reference answer and the underlying principle, while the pointwise prompt applies the full gate and component scheme. An answer that loses a head-to-head comparison on overall quality can still pass a gate the other answer failed.
We run pairwise when the question is “did this change help”, and pointwise when the question is “how good is this”. Why running both matters is covered in Dynamic Benchmarking: How We Did It, and the comparison code, compare_answers() in main.py, is in the Groundtruth Dynamic Benchmarking repository.