The problem: your documents aren't in any public benchmark
Ask "does this model know [topic]?" and a public benchmark can answer that. Ask "did this model learn anything from the 340 reports in our own archive?" and no public benchmark can, because it was never written against your documents in the first place. That gap matters most exactly when it's hardest to close: continual, corpus-specific fine-tuning, where the whole point is that the model should know your material better than a general model ever could.
General capability and corpus-specific knowledge acquisition are different questions, and a benchmark built for the first one can't answer the second.
How the generator works
Instead of hand-writing one benchmark, the pipeline is built to write benchmarks: point it at a new corpus and it produces a new exam from scratch, the same way every time.
- Read the corpus
- The pipeline goes through the documents you actually have, whether that’s exploration reports, an archive or a specialist literature, and builds an internal picture of what is actually testable in them, rather than starting from a fixed syllabus.
- Generate questions, answers and rubrics
- Each question is drafted with a reference answer and a grading rubric together, so the rubric is written against the same evidence the question came from, not reverse-engineered afterwards.
- Verify against the source
- Every claim in every reference answer is checked against a specific passage in the corpus before the question is accepted. A claim that can’t be traced back to the source doesn’t make it into the benchmark.
- Hold out the finished set
- The finished benchmark is kept private, so no model can train on the exact questions it will later be tested against. That’s the same contamination problem that eventually retires a static public benchmark.
The full authoring prompt behind these steps is public in AUTHORING.md, including the machine checks that run before a human ever reviews the output, covered in full in Generating a 50-Question Benchmark With One Approval Step.
Grading without one judge deciding everything
A model scoring its own homework, or a single judge with no check on its own bias, isn't rigorous grading. It's a second opinion with extra steps. Every question in a generated benchmark ships with a gate, one claim the question exists to test that zeroes the whole answer if missed, plus components that score the supporting reasoning on its own merits. The full rubric structure is covered in Dynamic Benchmarking: How We Did It.
None of that is trusted until it's tested. Every rubric is run against fabricated answers with a known, predicted score before it ever grades a real model, a step covered in Testing the Rubric Before You Test the Model. When an AI model does the grading, it needs the same scrutiny as the model being tested; what that scrutiny looks like in practice is the subject of LLM-as-a-Judge: Complete Guide to Using LLMs as Evaluators.
Proof of concept: geology
Geology is where we built and proved this first, as Groundtruth: real exploration reports and government geological archives, generated into a 50-question exam per source, graded by a judge that never sees the model being tested, with results on a public leaderboard. It's the proof that the pipeline holds up against real, messy, specialist documents rather than a clean toy corpus. It's not a boundary on what the pipeline can do.
Built for any specialist domain
We're releasing this as a geology benchmark generator because geology is the field we work in. But the underlying idea isn't geological. Any field with a specialist corpus, whether that's one exploration licence, one oil basin, one state's regulatory archive or one hospital system's records, can generate an exam tailored to what a model was actually meant to learn from it, using the same pipeline, the same calibration step, and the same verification-against-source discipline. The code, the corpus-to-rubric pipeline, and the worked geology example are MIT-licensed on GitHub, so the same process runs on a different body of documents.
Resources
Code
- GitHub - EigenformAI/groundtruth-dynamic-benchmarking
- AUTHORING.md - the full generation prompt
See it in action
- Groundtruth Dynamic Benchmark - the geology-specific benchmark this pipeline generates and grades
Reading
- Ground Truth: Dynamic Benchmarking in Highly Specific Fields - why a domain benchmark beats a public one
- Dynamic Benchmarking: How We Did It - the rubric structure in full
- Generating a 50-Question Benchmark With One Approval Step
- Testing the Rubric Before You Test the Model
- LLM-as-a-Judge: Complete Guide to Using LLMs as Evaluators
- How to Benchmark AI Models: A Practical Framework for Reliable Results
Frequently Asked Questions
+What is dynamic benchmarking?
A pipeline that reads a specific set of documents, such as one company’s archive, one region’s survey or one field’s literature, and generates a fresh exam from it: questions, reference answers and a calibrated grading rubric, verified against the source before any model is tested. It replaces a hand-authored benchmark with a benchmark generator.
+How do I do dynamic benchmarking?
At minimum: read your corpus and decide what a model would need to demonstrate to show it actually learned from it, not just general knowledge of the field. Draft each question with a reference answer and a rubric together, verify every claim in that answer against a specific passage in the source, then test the rubric itself on fabricated answers with a known score before grading anything real. Our own implementation of these steps, including the exact authoring prompt, is open source on GitHub. See AUTHORING.md for the exact prompt.
+How is this different from a static benchmark like MMLU?
A static benchmark is a fixed list of questions, useful until it leaks into training data and stops meaning anything. A dynamic benchmark is a reusable pipeline: point it at a new corpus and it produces a new, uncontaminated exam, so it never has to be retired the way a fixed question list does.
+Can I use this for a domain other than geology?
Yes. Geology is the domain we work in and the one we’ve proven this on with Groundtruth, but nothing about the pipeline is geological. The corpus-to-rubric generator, the calibration process and the scoring logic are all domain-agnostic and MIT-licensed on GitHub, so the same process runs on a different body of documents.
+How are answers graded without one judge deciding everything?
Every question ships with a gate (one claim that must be present, or the answer scores zero) plus components that award the remaining points, all calibrated in advance against fabricated answers with a known expected score. That calibration step is what catches a rubric that would otherwise double-count credit or pass the wrong reasoning, before it ever grades a real model.