Groundtruth Dynamic Benchmark

The benchmark for testing AI models and harnesses on geology.

Candidate models and agent harnesses answer a fixed set of questions taken from real exploration reports and government geological archives. A separate judge model scores each answer against the passage it came from, and the results sit on a public leaderboard.

GitHub ↗
The Groundtruth benchmark ranking: eight language models plotted by mean rubric score with 95% confidence intervals, split into tiers.
Models score from the public leaderboard

General benchmarks measure overall ability. Groundtruth is the one built specifically for geology: whether a model, or the harness wrapped around it, can read an actual exploration report or government geological filing and answer questions about what the document actually says, not what a model already knows about geology in general. The questions are not written by hand. They come from the same corpus-to-benchmark pipeline described in Dynamic Benchmarking, with every rubric line traced to a passage in the source and the finished set held private so no model can train on it.

The grading model is never the one being tested. It scores each answer from 0 to 10 against the rubric, either pointwise (one judge pass per question) or pairwise, judging two models’ answers together with each ordering used once so answer order doesn’t count. It also marks any answer that brings in material from outside the corpus. The rubrics are built so a high score is hard to fake. Each question has a pass/fail gate that must be cleared before any partial credit counts. The partial-credit points don’t overlap, some concepts are required for a full mark, and a do-not-credit list rules out common mistakes. The judge shows which parts of each answer earned credit.

Version 1.1.0 added the public leaderboard. Behind it are three 50-question sites, built from Western Australian exploration reports, USGS records, and NI 43-101 technical filings. A fourth, smaller site covers the Yudnamutana Copper district in South Australia, using 34 openly licensed deposit records (CC BY 4.0) as a worked example. If a model or a harness can be trusted on geology, this is the benchmark that shows it: not a general-knowledge quiz, but the actual documents the work runs on.

Why Groundtruth

The pipeline that generates and grades Groundtruth is domain-agnostic. The mechanics are covered in full in Dynamic Benchmarking. What makes Groundtruth specifically the benchmark for geology is what it's been pointed at:

Real coverage, not a toy corpus
Three 50-question sites built from Western Australian exploration reports (WAMEX), USGS supergene and hypogene ore studies, and NI 43-101 technical filings, plus a smaller openly licensed district as a worked example. This is the archive a geologist actually works from.
Two passes
Models and harnesses answer first. A different model grades second, so the thing under test never sees the rubric it will be judged against.
Scoring
Each answer is scored from 0 to 10 against the rubric. A run can also be pairwise, judging two answer sets against each other and using each ordering once so answer order doesn’t count. The judge flags any answer that pulls in material from outside the corpus, so a model can’t pad a geology answer with outside knowledge and still pass.
The rubrics
Every rubric is calibrated against worked examples of known quality before it grades a real model, so the pass/fail gate, the partial-credit thresholds, and the do-not-credit list all sit where a geologist would put them.

Public leaderboard

eigenform.ai/benchmark is a live LLM benchmark leaderboard for earth-science reasoning. It ranks current models and harnesses across the three 50-question sites, reading scores from the two Hugging Face datasets, and shows bootstrap confidence intervals and a per-section breakdown.

Resources

Code

  • GitHub · EigenformAI/groundtruth-dynamic-benchmarking

Leaderboard

Datasets

Frequently Asked Questions

+What is the best AI benchmark for geology?

That depends on what you need to measure, but Groundtruth is a public benchmark built specifically for geological reasoning: questions generated from real exploration reports and government geological archives rather than general-knowledge trivia, each answer graded against the source passage by an independent judge, with results on a public leaderboard and the grading methodology, including disclosed judge conflicts of interest, published in full.

+Does Groundtruth test agent harnesses, or only models?

Both. The model track ranks any model as submitted, in whatever harness its submitter used. A separate harness track pins the model to one reference id so a harness’s own contribution to the score can be measured on its own, independent of which model it happens to be running.

+Can a model just memorise the answers?

The finished question set is held private specifically so it can’t appear in training data, and the judge flags any answer that pulls in material from outside the corpus it was actually given. A model that already knows the general geology of a district still has to show it can read the specific documents in front of it.

+Where does the underlying corpus come from?

Three 50-question sites are built from Western Australian exploration reports (WAMEX), USGS supergene and hypogene ore studies, and NI 43-101 technical filings. A fourth, smaller site covers the Yudnamutana Copper district in South Australia using 34 openly licensed deposit records (CC BY 4.0), and ships in the repo as a worked example.

← Back to Eigenform