TL;DR

  • Standard benchmarks like IMO results or HumanEval work for general skills, but don’t tell you whether a model fine-tuned on a narrow, specific corpus - such as geological reports on one region of Western Australia - actually learned anything from that corpus.
  • The piece distinguishes capability benchmarks (“what can this model do”) from knowledge-acquisition benchmarks (“what did this model learn after reading this corpus”), arguing continual fine-tuning projects need the latter, which existing benchmarks conflate away.
  • Its proposed fix, “dynamic benchmarking,” replaces hand-authored benchmarks with a benchmark generator: a candidate model is examined by a panel of frontier models with corpus and web access that identify what to test, write questions and rubrics, and only then grade responses.
  • This is explicitly distinguished from LLM-as-judge: the examining panel doesn’t need to be smarter than the candidate, only better resourced and structured to reach a consensus standard.
  • Using the Cue Victory gold field as a case study, the piece separates three confounded things a naive benchmark would blur together - innate pretrained knowledge of a region, general data-analysis skill, and corpus-specific learning - and argues only the third is what a continual-learning benchmark should isolate.
  • The pipeline itself runs: read the corpus, build an internal understanding, identify testable concepts, generate questions and reference answers and rubrics, verify every answer against source evidence, then hold out the benchmark, with pairwise comparison against the untuned foundation model as the preferred scoring method.

Key Takeaways

  • The real contribution is a conceptual distinction most existing benchmarks blur: capability versus knowledge acquisition. That distinction only matters once you’re doing continual, corpus-specific fine-tuning rather than chasing general capability gains, which is why general benchmarks miss it.
  • The “exam board, not judge” framing is a structural design choice, not a euphemism: it doesn’t require graders smarter than the candidate, only ones better resourced and organised to converge on a standard. That is what makes the approach viable across specialist domains no single grader may fully understand on their own.
  • Treating benchmarks as reusable generation pipelines, not static artifacts, is presented as a durability strategy. Static benchmarks such as MMLU or GPQA decay once contaminated by training data; a pipeline can regenerate fresh questions for any new corpus instead of being retired.
  • The Cue Victory case study is treated as representative of a general measurement problem, not a geology-specific quirk. Separating pretrained knowledge, general skill, and corpus-specific learning is framed as a problem any narrow-domain fine-tuning benchmark has to solve.

Dynamic-Benchmarking-in-Highly-Specific-Fields

Eigenform dynamic benchmark: Ground Truth: Dynamic Benchmarking in Highly Specific Fields

Release notes: Release notes

How do you benchmark model performance when every deployment has its own corpus?

Obviously if you’re interested in getting a model to do maths you bencmark on IMO results, if you want it to learn to code you use HumanEval, and so on and so forth? But if you want to test its ability to navigate geological reports from a specific region of Western Australia – to pick an example out of the air – that’s much more difficult.

Sure, you can track its general geological skills, but that’s not very satisfying if you’re working on a continual fine-tuning project aiming to get it to expert status on this one specific topic.

Most benchmarks measure capability but we’re measuring knowledge acquisition, which is a much more fluid entity. Capability asks, “What can this model do?” Knowledge acquisition asks, “What did this model learn after reading this corpus?” Less “what are sedimentary rock” than “explain everything you have learnt about sedimentary rock formation”. A high schooler’s answer might be two sentences, a geology professor’s could be 20 pages. A good knowledge acquisition benchmark should be able to measure progress along this route rather than mere correctness.

This is where many current benchmarks fall down. Current benchmarks tell us whether Claude or Qwen knows geology in general. They do not tell us whether a local model has actually learned anything from our specific geological reports. And it’s not a pure epistemological nicety. A mining company doesn’t care whether the model knows about gold pathfinders sub speciae aeterinitatis. They care whether it learned them in Cue, WA.

Our solution to this is dynamic benchmarking. What exactly does this mean in practice?

Imagine training on:

  • one exploration licence
  • one oil basin
  • one state’s geological survey
  • one mine’s historical reports

You cannot realistically hand-author a benchmark for every new corpus. So instead of creating benchmarks you need to create a benchmark generator: your local show pony model acts as the candidate while a panel of frontier models with web and corpus access acts as the exam board: identifying what should be tested, writing questions, constructing grading rubrics, and only then evaluating responses.

It is important to emphasise that this is not the LLM-as-a-judge paradigm. Exam boards are not smarter than every candidate to face them, and they are structured in such a way as to account for the fact. They have better information access, an awareness of the abstractions being tested, and an ability to operate on a consensus basis, and they leverage that to set universal standards even for candidates who may have a far higher IQ than any of their members.

In our case we wanted to track a model’s ability to explore a data corpus, discover knowledge about a specific topic – in our case the Cue Victory gold field in Western Australia – and integrate that knowledge into its own weights via continual learning.

This is tricky to assess. All models have a certain level of knowledge of this region just by virtue of the fact that mining companies exploring in the area have been publishing reports on it for years and some of this information has got into the pre-training data. On the other hand, a skilled-but-naive data analysis model would be able to read through the corpus data and outperform a basic pre-trained model without corpus access.

In other words, there are several different things being measured here:

  1. Innate, weight-level knowledge of the Cue Victory site.
  2. Data analysis skills.
  3. Ability to navigate the Cue Victory dataset in particular (effectively a combination of the other two).

General benchmarks often conflate prior knowledge, reasoning ability and adaptation. For continual learning, we care specifically about the third. To accurately track our models’ improvement (or decline) we developed a dynamic benchmarking system that we are now open-sourcing. The basics of this system can be found in the skill.md document published to our Github, but the tldr is as follows:

  1. Read the corpus.
  2. Build an internal understanding.
  3. Identify concepts worth testing.
  4. Generate questions.
  5. Generate reference answers.
  6. Generate grading rubrics.
  7. Verify every answer against source evidence.
  8. Hold out the benchmark.

Obviously it’s not always that simple – you need to develop a relatively sophisticated and coherent schema to assess a “vague but basically ok” answer against a “detailed but with a couple of flubs” answer. You can check out the Github to see how we did this in practice.

You can then apply whatever scoring metric you like to the results as long as it’s reasonably rigorous. We like to compare finetuned models pairwise against the foundation model and see which one the panel of examiners prefers, but ymmv.

We’re releasing this as a geology benchmark because that’s the domain we work in. But we don’t think the underlying idea is geological. Any field with a specialist corpus should be able to generate an examination tailored to what the model was actually supposed to learn.

MMLU is an asset. GPQA is an asset. Humanity’s Last Exam is an asset. People build them once and then use them until question-answer pairs get into the training data and are replaced. We’re arguing for a formalisation of this process: not lists of questions but reliable pipelines for producing lists of questions.

The process gets stronger the more people adapt it for their own domains. Make your own from the benchmarking repo on GitHub.