Groundtruth Dynamic Benchmark

A test of how well a model reasons over real geological documents.

Candidate models answer a fixed set of questions taken from exploration reports and government archives. A separate judge model scores each answer against the passage it came from.

The Groundtruth benchmark ranking: eight language models plotted by mean rubric score with 95% confidence intervals, split into tiers.
Models score from the public leaderboard

General benchmarks measure overall ability. Groundtruth is narrower. It checks whether a model can read an actual exploration report or geological filing and answer questions about what the document says. The questions are not written by hand. A pipeline goes through the corpus, works out what is worth testing, and drafts each question together with a reference answer and a grading rubric. Every rubric line points to a passage in the source. The finished set is kept private so no model can train on it.

The grading model is never the one being tested. It scores each answer from 0 to 10 against the rubric, either pointwise (one judge pass per question) or pairwise, judging two models’ answers together with each ordering used once so answer order doesn’t count. It also marks any answer that brings in material from outside the corpus. The rubrics are built so a high score is hard to fake. Each question has a pass/fail gate that must be cleared before any partial credit counts. The partial-credit points don’t overlap, some concepts are required for a full mark, and a do-not-credit list rules out common mistakes. The judge shows which parts of each answer earned credit.

Version 1.1.0 added the public leaderboard. Behind it are three 50-question sites, built from Western Australian exploration reports, USGS records, and NI 43-101 technical filings. A fourth, smaller site covers the Yudnamutana Copper district in South Australia. It uses 34 openly licensed deposit records (CC BY 4.0) and ships in the repo as a worked example. The code is MIT-licensed, and the corpus-to-rubric pipeline is included, so you can point it at a different body of documents.

How it works

Writing the benchmark
The pipeline reads the corpus and works out what to test, then writes each question with its reference answer and rubric. Answers are checked against the source before the set is held back.
Two passes
Models answer first. A different model grades second, so the model under test never sees the rubric.
Scoring
Each answer is scored from 0 to 10 against the rubric. A run can also be pairwise, judging two answer sets against each other and using each ordering once so answer order doesn’t count. The judge flags answers that use material from outside the corpus.
The rubrics
Every rubric is calibrated against worked examples of known quality, so the pass/fail gate, the partial-credit thresholds, and the do-not-credit list all sit where they should.

Public leaderboard

eigenform.ai/benchmark is a live LLM benchmark leaderboard for earth-science reasoning. It ranks current models and harnesses across the three 50-question sites, reading scores from the two Hugging Face datasets, and shows bootstrap confidence intervals and a per-section breakdown.

Resources