TL;DR
- Standard benchmarks work for general skills, but they can’t tell you whether a model fine-tuned on a narrow corpus, such as geological reports on one region of Western Australia, actually learned anything from it.
- Dynamic benchmarking replaces hand-authored benchmarks with a benchmark generator, the approach behind Groundtruth, Eigenform’s open-source benchmark for geological reasoning. A panel of frontier models with corpus and web access acts as the exam board: it decides what to test, writes questions and rubrics, and only then grades the candidate.
- The pipeline reads the corpus, identifies testable concepts, generates questions, reference answers and rubrics, verifies every answer against source evidence, and holds the benchmark out. Pairwise comparison against the untuned foundation model is the preferred way to score it.
Key Takeaways
- Separate capability from knowledge acquisition, which most existing benchmarks blur. That distinction only matters once you’re doing continual, corpus-specific fine-tuning rather than chasing general capability gains, which is why general benchmarks miss it.
- An exam board doesn’t need graders smarter than the candidate, only ones better resourced and organised to converge on a standard. That is what makes the approach viable across specialist domains no single grader may fully understand on their own.
- Treat benchmarks as reusable generation pipelines, not static artifacts. Static benchmarks such as MMLU or GPQA decay once contaminated by training data; a pipeline can regenerate fresh questions for any new corpus instead of being retired.
This post explains dynamic benchmarking in highly specific fields, the idea behind our open-source Groundtruth benchmark.
How do you benchmark model performance when every deployment has its own corpus?
Obviously if you’re interested in getting a model to do maths you benchmark on IMO results, if you want it to learn to code you use HumanEval, and so on and so forth? But if you want to test its ability to navigate geological reports from a specific region of Western Australia (to pick an example out of the air), that’s much more difficult.
What Is Dynamic Benchmarking?
Dynamic benchmarking in highly specific fields means generating the benchmark from the corpus a model should learn. It replaces a fixed test. The questions, reference answers and grading rubrics all come from that corpus. So the score measures knowledge acquisition: what the model learned from the material, not its general capability.
For the geological reports above, that means a benchmark built from those reports, not a general geology exam. Sure, you can track its general geological skills, but that’s not very satisfying if you’re working on a continual fine-tuning project aiming to get it to expert status on this one specific topic.
What Is the Difference Between Capability and Knowledge Acquisition?
Most benchmarks measure capability but we’re measuring knowledge acquisition, which is a much more fluid entity. Capability asks, “What can this model do?” Knowledge acquisition asks, “What did this model learn after reading this corpus?” Less “what are sedimentary rock” than “explain everything you have learnt about sedimentary rock formation”. A high schooler’s answer might be two sentences, a geology professor’s could be 20 pages. A good knowledge acquisition benchmark should be able to measure progress along this route rather than mere correctness.
The two can diverge. A LoRA fine-tune on a narrow corpus can improve in two ways that don’t imply each other. It can get better at geology as a discipline. Or it can just get better at reciting the facts in its own training data. Its general reasoning may not improve at all, or may even regress. For a fine-tuned model, the useful question compares it with the untuned base model. What does it now know about this material that it did not know before?
Why Do General Benchmarks Fail in Highly Specific Fields?
This is where many current benchmarks fall down. Current benchmarks tell us whether Claude or Qwen knows geology in general. They do not tell us whether a local model has actually learned anything from our specific geological reports. And it’s not a pure epistemological nicety. A mining company doesn’t care whether the model knows about gold pathfinders sub specie aeternitatis. They care whether it learned them from the WAMEX reports on their own ground.
None of this makes MMLU or GPQA bad benchmarks. They answer a different question: how capable is a model across broad subjects? They answer it well. But they cannot separate knowledge a model brought with it from knowledge it acquired from your reports. Nobody wrote their questions about your corpus. They test existing knowledge, while a fine-tune needs a test of newly acquired domain knowledge. Specialist industries care about one company’s own ground. They need dynamic benchmarking in highly specific fields, an evaluation built around that material.
How Does Groundtruth Enable Dynamic Benchmarking?
Our solution to this is dynamic benchmarking. What exactly does this mean in practice?
Imagine training on:
- one exploration licence
- a single oil basin
- a state’s geological survey
- the historical reports of one mine
You cannot realistically hand-author a benchmark for every new corpus. So instead of creating benchmarks you need to create a benchmark generator: your local show pony model acts as the candidate while a panel of frontier models with web and corpus access acts as the exam board: identifying what should be tested, writing questions, constructing grading rubrics, and only then evaluating responses.
Groundtruth is our implementation of that benchmark generator. It starts from one highly specific corpus and identifies the concepts worth testing. Then it generates questions, reference answers and grading rubrics. The resulting benchmark evaluates the candidate model. Its first edition uses geology reports, because geology is the domain we work in.
An exam board, not an LLM judge
It is important to emphasise that this is not the LLM-as-a-judge paradigm. Exam boards are not smarter than every candidate to face them, and they are structured in such a way as to account for the fact. They have better information access, an awareness of the abstractions being tested, and an ability to operate on a consensus basis, and they leverage that to set universal standards even for candidates who may have a far higher IQ than any of their members.
Three things a corpus benchmark has to separate
In our case we wanted to track a model’s ability to explore a data corpus, discover knowledge about a specific topic (in our case one Western Australian goldfield, the WAMEX district) and integrate that knowledge into its own weights via continual learning.
This is tricky to assess. All models have a certain level of knowledge of this region just by virtue of the fact that mining companies exploring in the area have been publishing reports on it for years and some of this information has got into the pre-training data. On the other hand, a skilled-but-naive data analysis model would be able to read through the corpus data and outperform a basic pre-trained model without corpus access.
In other words, there are several different things being measured here:
- Innate, weight-level knowledge of the WAMEX site.
- Data analysis skills.
- Ability to navigate the WAMEX dataset in particular (effectively a combination of the other two).
General benchmarks often conflate prior knowledge, reasoning ability and adaptation. For continual learning, we care specifically about the third. To accurately track our models’ improvement (or decline) we developed a dynamic benchmarking system that we are now open-sourcing.
How Does Groundtruth Generate a Benchmark From a Custom Corpus?
To benchmark a fine-tuned model on a custom corpus, Groundtruth runs eight steps. Our SKILL.md document on GitHub codifies them: they turn the corpus into questions, reference answers and grading rubrics.
1. Read the Corpus
Read the corpus and inventory it first. Note the source families, terminology, duplicated material, contradictions and OCR variants. You need to know where the corpus is too thin to assess before you commit a question there.
2. Build an Internal Understanding
Build an internal understanding of what the corpus says and which parts to trust. Direct primary text outranks OCR, and OCR outranks an auto-generated summary. Where sources disagree, resolve by authority or keep both with attribution. Never merge them into a fictitious compound fact.
3. Identify Concepts Worth Testing
Identify the concepts worth testing. For each one, distill the principle behind it. A right answer then has to show the reasoning, not just match a sentence in the source.
4. Generate Questions
Generate the questions first, in their own pass, without answers or rubrics in view. Ideally a separate agent writes them. That stops anyone reverse-engineering the marking scheme to fit a pre-written answer.
5. Generate Answers
Then generate reference answers. Each one cites a specific book, page and line in the source. A correct answer then traces to a real passage, not a plausible paraphrase.
6. Generate Rubrics
Then generate the grading rubrics. Each question gets one hard gate, worth 2 to 4 of the 10 points. Miss the gate and the answer scores zero. Pass it, and non-overlapping graduated components award the rest.
Obviously it’s not always that simple. You need to develop a relatively sophisticated and coherent schema to assess a “vague but basically ok” answer against a “detailed but with a couple of flubs” answer.
7. Verify
Verify every answer against source evidence. A final verifier checks questions and answers against the corpus. Every question also ships with test responses and their expected scores. An independent grader must reproduce those scores from the rubric alone.
8. Hold Out the Benchmark
Hold out the benchmark. Keep it out of the candidate model’s training data. Then a score reflects what the model learned, not the test itself.
Scoring the results
You can then apply whatever scoring metric you like to the results as long as it’s reasonably rigorous. We like to compare finetuned models pairwise against the foundation model and see which one the panel of examiners prefers, but ymmv.
Static vs Dynamic Benchmarks: What Is the Difference?
In our usage, a static benchmark is a fixed set of questions and answers. A dynamic benchmark grows out of the corpus under test. Static benchmarks are consistent, because every model meets the same test. That is what you want for general evaluation. Dynamic benchmarking in highly specific fields answers a different question. Did a model learn a specific, possibly changing body of knowledge? When the corpus changes, you regenerate the test.
| Static benchmark | Dynamic benchmark | |
|---|---|---|
| Questions | Fixed, written once | Generated from the corpus under test |
| Best for | Consistent general evaluation | Whether a model learned a specific corpus |
| Main risk | Decays once its questions leak into training data | Only as good as its generation and verification |
MMLU is an asset. GPQA is an asset. Humanity’s Last Exam is an asset. People build them once and then use them until question-answer pairs get into the training data and are replaced. We’re arguing for a formalisation of this process: not lists of questions but reliable pipelines for producing lists of questions.
Can Dynamic Benchmarking Be Used Beyond Geology?
We’re releasing this as a geology benchmark, with results on the Groundtruth leaderboard, because that’s the domain we work in. But we don’t think the underlying idea is geological. Any field with a specialist corpus should be able to generate an examination tailored to what the model was actually supposed to learn.
Dynamic benchmarking in highly specific fields carries over to other domains. The pipeline is domain-agnostic: it needs a corpus, a way to read it and a panel to set the exam. A law firm could run the same steps on its contracts. So could a hospital with clinical guidelines, or a manufacturer with maintenance records. The shipped authoring skill targets geology, so you would adapt its question rules, such as spatial and stratigraphic scope. The common requirement is a clearly defined body of knowledge that the model should learn.
The process gets stronger the more people adapt it for their own domains. Make your own from the benchmarking repo on GitHub.
FAQs
What does dynamic benchmarking measure?
It measures knowledge acquisition: what a model learned from a specific corpus, rather than its general capability. The questions, reference answers and rubrics all come from that corpus. So a model only scores well if it knows the material. Comparing a fine-tuned model with its untuned base model shows how much of that knowledge came from training.
What is Groundtruth?
Groundtruth is Eigenform’s open-source (MIT) benchmark for geological reasoning and our implementation of dynamic benchmarking. Its pipeline reads a corpus and generates questions, reference answers and rubrics. It verifies them against the source. A judge then grades each candidate model and shows which rubric components it credited. Results for every model we have run are on the Groundtruth leaderboard.
How do you benchmark a fine-tuned model on a custom corpus?
Generate the benchmark from the corpus itself. Read it, identify the concepts worth testing, and write questions, reference answers and rubrics. Verify each answer against source evidence, and keep the set out of training. Then run the fine-tuned model and its untuned base model through it. A judge compares their answers pairwise, in both orders.
How is knowledge acquisition different from capability?
Capability is what a model can already do; knowledge acquisition is what it learned from a target corpus. A general benchmark measures the first. A fine-tune on a narrow corpus can improve at the second while its general scores stay flat. It can also learn to recite its training text without reasoning better. So the two need separate measurements.
