TL;DR

  • Announces v1.0.0 of the Groundtruth Dynamic Benchmarking System on GitHub: not a fixed set of benchmark problems, but a pipeline for generating reliable benchmark questions and scoring rubrics for whatever corpus you’re working with.
  • Ships a working example grounded in a real, openly licensed corpus - 34 mineral deposit records from the Yudnamutana Copper district, South Australia - runnable end-to-end with a single script after cloning the repo and adding API keys.
  • Uses a rubric format built to resist grade inflation: a hard gate per question, non-overlapping graduated components, required concepts, and do-not-credit lists, with the underlying mechanics covered in the companion post “Dynamic Benchmarking: How We Did It.”
  • Grading is structured and auditable: the judge returns a full breakdown of which components it credited and why, and flags anything it can’t verify from the rubric alone rather than guessing.
  • The corpus-to-rubric authoring pipeline itself ships in the repo, so anyone can point it at their own corpus and build their own benchmark edition.

Key Takeaways

  • The release is framed as a generalisable tool rather than a geology deliverable. The geology example exists to demonstrate a domain-agnostic pipeline, not because the benchmark’s value is specific to that field.
  • Auditability is treated as a first-class feature, not an afterthought. A judge that shows its reasoning and flags what it can’t verify is a different, more trustworthy thing than one that returns a bare score.
  • The actual bet is on the pipeline being reused, not on the example benchmark being adopted. Open-sourcing the authoring skill itself, rather than only its output, is what lets the value compound as more people adapt it to their own corpora.

Groundtruth Dynamic Benchmarking (Geology Edition) v1.0.0 Is Live

Eigenform dynamic benchmark: Ground Truth: Dynamic Benchmarking in Highly Specific Fields

Release notes: Release notes

The first runnable release our Groundtruth Dynamic Benchmarking System is live on GitHub. The core concept is simple: rather than offering a suite a benchmarking problems, we offer a pipeline for generating reliable benchmark problems and scoring rubrics for whatever problem you’re working to solve.

What you get

  • A working example, ready to run. The sample rubric is grounded in a real corpus: 34 mineral deposit records from the Yudnamutana Copper district, South Australia, openly licensed under CC BY 4.0 AU. You don’t need to track down source material yourself. Clone the repo, fill in your API keys, and ./start_eval.sh runs end to end.
  • A rubric format built to resist grade inflation. A hard gate per question, non-overlapping graduated components, required concepts, and do-not-credit lists. The mechanics, and the problems that shaped them, are covered in Dynamic Benchmarking: How We Did It.
  • Structured, auditable grading. The judge returns a full breakdown of which components it credited and why, instead of a single number. It flags anything it can’t verify from the rubric alone rather than guessing.
  • The authoring skill itself. The same corpus-to-rubric pipeline we used to build the geology edition ships in the repo, so you can point it at your own corpus and build your own edition.

Try it now

bash 
git clone https://github.com/EigenformAI/groundtruth-dynamic-benchmarking
cd groundtruth-dynamic-benchmarking
cp .env.example .env   # fill in the keys you need
uv sync
./start_eval.sh

Read more