TL;DR
- We built a benchmark to check whether a LoRA fine-tuned on regional geology actually reasons better about geology, or just recalls its own training data better.
- A large model generated questions from historical tenement reports, each traceable to a source passage. Grading criteria come from the underlying geological principle, not the source text, and a geology expert vetted every question and rubric.
- Five problems shaped the final design: judges that reward fluency, source material the base model already knows, pointwise and pairwise scores that disagree, generations that fail outright, and run-to-run noise.
Key Takeaways
- Design the “learned vs memorised” distinction into the rubric. Deriving grading criteria from the principle rather than the source sentence is what makes a right answer demonstrate reasoning instead of paraphrase.
- Close each loophole an LLM judge would reward with its own structural fix. Hard gates, non-overlapping components, contradiction rules and swapped pairwise comparisons each target one failure mode; a general appeal to “better judgment” targets none.
- Treat operations as part of the benchmark. Obscure sources, capped retries and repeated runs with reported spread decide whether you can trust a result as much as the rubric does.
This is part of a broader series on building specialist models: fine-tuning a base model for a narrow technical domain, then checking whether the fine-tune actually earned its keep. That check is fine-tuned model evaluation, and it needs a different benchmark from the general-purpose ones.
This post is how we built ours. The code is in the Groundtruth Dynamic Benchmarking repo, whose first runnable release was v1.0.0.
Did the fine-tune learn geology, or memorise it?
We fine-tuned a LoRA adapter to specialise a base model in regional geology, specifically the historical exploration reports behind our WAMEX district, from one Western Australian goldfield. After training, the harder question was whether it actually made the model better at geology, and specifically at the geology of this region, or just better at sounding like it.
This is an account of the benchmark we built to answer that question. We frame it as the problems we had to solve, roughly in the order we hit them, in case you’re building the equivalent for your own domain. The same method now runs the public Groundtruth benchmark and its leaderboard.
How our fine-tuned model evaluation works
Before getting into what went wrong, here’s the shape of the benchmark end to end.
Two things can count as “got better”
A LoRA fine-tune trained on a narrow corpus can improve in two ways that don’t imply each other. It can get better at geology as a discipline, applying the principles a geologist would bring to material it’s never seen. Or it can just get better at reciting the specific facts in its own fine-tuning data, with no improvement (or even a regression) in general reasoning. Only the first kind is useful, so the benchmark had to tell them apart rather than collapse them into one score.
We didn’t build two separate test sets for that. It’s built into how each question is generated (more under “Building the grading criteria” below): the checkable criteria come from an explicit geological principle rather than the surface text, so a right answer has to demonstrate the reasoning, not just match a sentence in the source.
Building the question set
We picked source material from a set of historical tenement reports for that goldfield. A large closed-source model read through the documents and generated a fixed set of questions with reference answers, each one citable back to a specific book/page/line in the source, so a correct answer traces to a real passage rather than a plausible-sounding paraphrase. This produced 35 questions, a set we expanded in later iterations. This corpus-to-questions-to-rubric process is the one we later codified as a runnable skill; see How we build the questions and rubric below.
Building the grading criteria
For each question, we had the same generating model work top-down: distill the geological principle underpinning the answer, then derive the marking scheme from that principle rather than reading it off the page. The structure is a hard gate for minimum correctness, then graduated components above it (detailed in the section below). None of this shipped un-reviewed.
A human geology expert vetted every model answer and grading criteria before it went into the fixed set, so a mistaken principle or a wrong gate from the generating model couldn’t quietly become the standard the fine-tune was judged against.
Grading the fine-tune’s answers
We ran the fine-tuned model against that fixed question set and had a model score each answer pointwise against the rubric from the previous step: gate, then components, then required concepts, then do-not-credit, summed and capped at 10 out of 10.
Pairwise against the untrained baseline
A pointwise score alone doesn’t tell you whether the fine-tune actually moved the needle relative to where it started, so we also ran the same fixed question set through the untrained base model and had the judge compare the two answers head-to-head, question by question. We run that comparison twice with the order of the two answers swapped and reconcile the verdicts, so the result reflects which answer is better rather than which one the judge happened to see first.
How we build the questions and rubric
The rubric is where a benchmark lives or dies, so we build it as a deliberate, staged process rather than a single “write a marking scheme” prompt. We’ve since codified the whole thing as a runnable skill, build-source-grounded-groundtruth-benchmarking-geology, that turns a supplied corpus into questions, reference answers, a machine-readable rubric, and validation evidence. The shape of it:
Explore the corpus first
Inventory the source families, terminology, duplicated material, contradictions, and OCR/transliteration variants first. You can’t write a fair question about material you haven’t mapped, and you need to know where the corpus is too thin to assess before you commit to a question there.
Write a source-authority contract
Not all files are equally trustworthy: direct primary text outranks OCR, which outranks an auto-generated summary or a filename. When sources disagree, decide whether they actually conflict or just address different scales, dates, or scopes, and either resolve by authority or preserve both with attribution. Never merge two datasets into a fictitious compound fact.
Separate question-writing from answer-writing
We draft questions first, in their own pass, without their answers or rubrics in view, ideally by a separate agent. This keeps the marking scheme from being quietly reverse-engineered to fit a pre-written answer. A final verifier then adjudicates both against the corpus.
The prompt spells out what a question is allowed to be before it’s admitted. An excerpt:
Each question must:
- Be answerable under the declared candidate-access mode.
- Test one coherent construct, even if it contains linked subparts.
- Use exact source semantics and the correct spatial, temporal, stratigraphic, or administrative scope.
- Define counting and deduplication rules where a numeric answer could otherwise vary.
- Distinguish observations from interpretations and targets from discoveries.
- Avoid hidden retrieval requirements that appear only in the rubric.
- Avoid giving away the answer through excessive context.The full prompt is the build-source-grounded-groundtruth-benchmarking-geology skill.
Score 0–10 through one hard gate, then graduated credit
Each question has exactly one gate (component C1, worth 2–4 points): the minimum central correctness the answer must establish. Miss the gate and the score is zero: no peripheral credit leaks through. Pass it, and the remaining components (non-overlapping, integer points, totalling exactly 10) award demonstrated reasoning positively. Omissions simply go unrewarded; the rubric never penalises them.
Keep keywords as aids, not marks
Required concepts summarise what a component is testing, but exact wording never earns points on its own. A reasoning component demands a demonstrated causal or comparative link, not keyword presence. Accepted-variant and do-not-credit lists catch valid paraphrases and plausible-but-wrong answers respectively.
Make every marking block self-contained
A grader holding one question, one answer, and that question’s block must be able to score it without opening the corpus, an appendix, or another question. The block repeats the local principle and caveats wherever needed rather than referencing them.
Calibrate before trusting it
Every question ships with at least three test responses: a gate-fail near-miss (expected 0), a minimum gate pass (expected = C1), and a strong response (expected 9–10). An independent grader must reproduce those scores from the block alone. If it can’t, we rewrite the block.
Five problems we hit along the way
That is the process when it goes cleanly. Five problems surfaced getting it there, listed here roughly in the order we hit them.
Judges reward fluency over correctness
Give an LLM judge a loose “score this out of 10” instruction and it will over-reward answers that sound like the domain, all the right vocabulary and confident tone, whether or not they’re actually correct. You need a rubric a model can’t talk its way past. At minimum:
- A hard gate per question: the one fact the question is actually testing. Miss it and the score is zero, no matter how much correct-but-beside-the-point material surrounds it.
- Graduated, non-overlapping components above the gate, so nothing gets credited twice and extra unrequested material can’t push the score past a fixed ceiling, even if it’s correct.
- An all-or-nothing rule for binary facts: determinate data like a single name, number, or structural orientation must be scored as binary: full points or zero, with no partial credit for guessing around them.
- A rule that reasoning components require demonstrated logic, not required-concept presence, so a model can’t hit every keyword and still explain nothing.
- A strict contradiction rule: an answer asserting a claim and its negation earns zero for that component, countering the model’s tendency to hedge by providing multiple conflicting answers.
- A do-not-credit list of plausible wrong answers per question, so a common misconception doesn’t get rewarded just for sounding fluent.
Our rubric format and judge prompt already implement this: gate, then components, then required concepts, then do-not-credit, summed and capped. None of it is geology-specific; swap in your own question set and it holds.
The base model may already know your sources
If general pretraining knowledge can answer your questions, you’re benchmarking the base model, not your fine-tune. We used obscure historical tenement reports for Western Australia, the WAMEX district. They’re obscure enough that a correct answer requires those specific documents, not general geology.
Whatever you pick, expect it to be messier than you’d like unless it’s a clean, professionally-edited text. The same proper noun spelled three ways, numbers that disagree from one section to the next, OCR garbled enough to flip a value. Resolving that is on you: deciding which of the source’s own contradictions counts as correct, and building a synonym table so the judge doesn’t penalise a right answer for a spelling variant. This is the one problem here that no tooling solves for you, ours included.
Pointwise and pairwise scores can disagree
A model can tie a baseline on average pointwise score and still lose more head-to-head comparisons, or the reverse. Run only one mode and you’re working from half the picture. Pairwise also needs the comparison run twice with the answer order swapped and reconciled, or the result carries a position bias that has nothing to do with which answer is actually better. (We modelled our evaluation logic closely on enterprise standards, such as those outlined in Google’s agent evaluation metrics).
Our scoring script runs both modes with the swap-and-reconcile step already built in. Doing it yourself means roughly doubling your judge-call budget per question, and settling on a reconciliation rule (we average the two swapped verdicts) before you start, not after you’ve already got numbers that don’t agree.
Some generations fail outright
A model under test will occasionally hand back an empty completion, a truncated fragment, a wall of leaked reasoning tokens with nothing usable inside, or a flat refusal.
You still need a cleaning step that strips known leaked-token patterns before anything gets judged, so a real answer buried in scaffolding isn’t penalised for the scaffolding. When the cleaned output is genuinely empty, we retry the question on a fresh pod up to three times. Most of these failures are transient, an opencode process that died or a pod that hiccupped, and a retry usually gets a real answer out of the model.
The line is that we cap the retries. We log a question that fails all three attempts as an error and never count it as a success, rather than sweeping it under an unbounded retry loop. A model that cannot reliably produce an answer is just as much a result as one that produces the wrong answer. At scoring time we retry nothing: an empty answer that reaches the judge fails the hard gate and scores a zero. You measure exactly what the model produced, failures and all.
A single run is noise
Sampling variance alone can move a 25- or 35-question benchmark by a point or more between runs. Run each question several times per model and report the average, ideally with the spread, instead of treating one pass as the answer. A single number can’t tell you whether a difference is real or just noise.
