TL;DR

  • Groundtruth Dynamic Benchmarking v1.0.0 is on GitHub. It is not a fixed set of benchmark questions but a pipeline that generates reliable questions and scoring rubrics for whatever corpus you work with.
  • The release includes a runnable example built on 34 openly licensed mineral deposit records from the Yudnamutana Copper district, South Australia. Clone the repo, add API keys, and one script runs it end to end.
  • Grading is auditable: the judge shows which components it credited and why, and flags anything it can’t verify from the rubric instead of guessing.

Key Takeaways

  • The geology edition demonstrates a domain-agnostic pipeline. The same authoring skill ships in the repo, so any field with a specialist corpus can build its own edition.
  • A judge that shows its reasoning is more trustworthy than one that returns a bare score. Component-level credit and explicit “can’t verify” flags make every grade checkable.
An old geological map spread on a wooden desk under a lamp, its right half overlaid with a glowing blue grid and a floating label reading RELEASE: v1.0.0, STATUS: PUBLISHED, BUILD: STABLE

The first runnable release of our Groundtruth Dynamic Benchmarking System is live on GitHub, and the release notes cover everything in v1.0.0 in detail. The core concept is simple: rather than offering a suite of benchmarking problems, we offer a pipeline for generating reliable benchmark problems and scoring rubrics for whatever problem you’re working to solve, because knowledge acquisition needs its own kind of benchmark.

What Groundtruth Dynamic Benchmarking v1.0.0 gives you

  • A working example, ready to run. The sample rubric is grounded in a real corpus: 34 mineral deposit records from the Yudnamutana Copper district, South Australia, openly licensed under CC BY 4.0 AU. You don’t need to track down source material yourself. Clone the repo, fill in your API keys, and ./start_eval.sh runs end to end.
  • A rubric format built to resist grade inflation. A hard gate per question, non-overlapping graduated components, required concepts, and do-not-credit lists.
  • Structured, auditable grading. The judge returns a full breakdown of which components it credited and why, instead of a single number. It flags anything it can’t verify from the rubric alone rather than guessing.
  • The authoring skill itself. The same corpus-to-rubric pipeline we used to build the geology edition ships in the repo, so you can point it at your own corpus and build your own edition.

Try the Groundtruth Benchmark now

The Groundtruth benchmark measures geological reasoning over real source reports, and results from every model we have run are on the leaderboard. To run it yourself:

bash 
git clone https://github.com/EigenformAI/groundtruth-dynamic-benchmarking
cd groundtruth-dynamic-benchmarking
cp .env.example .env   # fill in the keys you need
uv sync
./start_eval.sh