TL;DR

  • Argues that standard academic AI benchmarks are stale (“an exam paper everyone has had access to for years”) and largely irrelevant to a real decision like spending $100,000 integrating an AI system into an exploration workflow, since geology is hyper-local and a general fact, such as bismuth being a gold pathfinder, may not hold in a specific terrain.
  • Introduces the Groundtruth Dynamic Benchmark, which generates fresh questions and scoring rubrics directly from a user’s own document set rather than testing against a fixed, generic question bank.
  • Offers three free sample test sets (Murchison WAMEX gold reports, USGS Southwestern US reports, Canadian technical reports) alongside support for fully custom datasets, with generated questions spanning four types - finding facts, reconstructing sequences of geological events, reconciling sources that disagree, and recognising when the evidence doesn’t support a firm answer - graded against a rubric rather than accepted for sounding persuasive.
  • Reports the results of a six-model bake-off: Kimi K3 scored highest overall, but confidence intervals overlap enough among the top three models that the benchmark can only reliably sort models into tiers rather than a strict ranking, and that tiering held consistently across different source-document types.
  • Given the near-tied top performance, shifts the comparison to cost: generation cost dwarfs judging cost and varies sixfold between models doing the same task, and plotting price against performance puts four models on a shared frontier of roughly equivalent return on cost - until a newly released model, Ox Alpha, is added, whose free-tier pricing puts it alone on a sharply shifted frontier ahead of everything else.
  • Concludes with the practical payoff: an exploration manager can benchmark any model-plus-retrieval or model-plus-agent combination against their own frozen, company-specific test set, and rerun that same benchmark later to check whether a proposed cheaper migration actually degrades performance on their own work, rather than relying on vendor claims or generic scores.

Key Takeaways

  • The central argument is that benchmark relevance, not benchmark existence, is the real problem in specialist domains. A high score on a generic geology benchmark says almost nothing about whether a model will work on a specific company’s own archive, which is why the tool generates a new benchmark per dataset rather than reusing one.
  • Reporting confidence intervals rather than a bare ranking is treated as the honest reading of the data. Given the overlap among the top three models, presenting “Kimi K3 won” without the tiering caveat would overstate what the benchmark actually shows.
  • Once accuracy differences between top models become statistically indistinguishable, cost becomes the deciding factor. That is why the piece treats the price-performance frontier, not the leaderboard, as the more actionable output of the exercise.
  • The benchmark’s real long-term value is framed as a standing tool for re-evaluating vendor claims over time, rather than a one-time model selection exercise. Rerunning the same frozen benchmark against a proposed cheaper model is only possible because the benchmark is generated from the buyer’s own data rather than a shared public one.

If you gave an AI the same messy exploration archive you would give a consulting geologist, how much would you trust what came back?

Traditional AI benchmarks are basically an exam paper everyone has had access to for years. Eventually you don’t really know whether a model has learned the subject or learned the exam. This isn’t the only problem when it comes to working out which model is best for geological work, however.

Suppose you’re an exploration manager with 15 years of reports, logs, maps, assays and consultant studies sitting in SharePoint. Someone wants you to spend $100,000 integrating an AI system. The vendor tells you that Model X scored 83% on some academic geology benchmark. You say: who cares?

Because every terrain is so unique, it can be actively damaging to rely too much on general knowledge. Sure, bismuth is often a gold pathfinder, but if you’re an explorer you want to know whether it’s a pathfinder in your specific geology.

That’s why we made the Groundtruth Dynamic Benchmark. Rather than asking a standard set of questions to test the sector knowledge of a model, Groundtruth generates a fresh set of questions and scoring rubrics from your own dataset, letting you compare models and agent harnesses on the data that matters to you.

How it Works

If you want to test the skill and versatility of a new model or agent harness, we provide three diverse test sets free: a set of Wamex reports from a gold mining property in the Murchison, four historical USGS reports on the Southwestern US, and four Canadian technical reports. If you want to compare model or agent skills on your own data, the system will generate a custom question set and scoring rubric based purely on that dataset. The generation process is set up such that some will require finding facts. Others will require reconstructing sequences of geological events, distinguishing competing interpretations or deciding what the evidence actually supports.

Can it find the evidence? “Which formations make up this succession?”

Can it reconstruct what happened? “What happened first: metamorphism, deformation or gold mineralisation?”

Can it reconcile geologists who disagree? “One report says there is a major shear. Another says there isn’t. What does the evidence actually establish?”

Does it know when the evidence runs out? “Two reports use different names for apparently similar greenstone belts. Can the model resist simply declaring them identical?”

Once you have your data and your question set, you’re ready to test. Pick a model and add in a custom agent harness (if you want one) then let it run. Once complete, the grading AI will check the answers - making sure that they match the grading rubric rather than simply being persuasive-sounding verbiage.

So Who Won?

Kimi K3.

Model scores by tier, with a 95% bootstrap confidence interval per question, split out by district
Tiers, not ranks: overlapping confidence intervals mean the benchmark can only reliably order models into tiers, not exact ranks - and the per-district breakdown (Technical / WAMEX / USGS) shows those tiers hold up across different source material

But that’s not the full story.

While Kimi got the highest one-off score, statistically there was enough overlap between the three top models that they can more or less be treated as equally good. This being so, the question of price is an important one.

Generation cost per model across the full question set, and cost per question on a log scale
What it cost: generation dwarfs judging, and varies six-fold between models doing the same task

Price plotted against performance for six models, with the Pareto frontier connecting Deepseek V4 Pro, GPT 5.6 Sol, Claude Sonnet 5 and Kimi K3
Price and performance before Ox Alpha: four models sit on the Pareto frontier, offering roughly equivalent return on cost

If we compare price and accuracy, we end up with four models offering roughly equivalent ROI: Deepseek V4 Pro, GPT 5.6 Sol, Claude Sonnet 5 and Kimi K3… at least until Ox Alpha was released, which shifted the Pareto frontier drastically.

Price plotted against performance with Ox Alpha added, sitting alone on a sharply shifted Pareto frontier
Price and performance after Ox Alpha: its free-tier pricing moves it onto the frontier by itself, ahead of every other model on cost

What it Means

This means that the exploration manager from the opening paragraphs can now run:

  • Commercial models + vanilla retrieval
  • Commercial models + their proposed RAG system
  • Commercial models + an agentic search harness
  • Whatever the vendor is proposing

against a custom benchmark derived from their own documents.

Then six months later, when someone says “we should migrate to Model Y because it’s 40% cheaper,” they can rerun the same frozen company benchmark and find out whether that change actually degrades performance on their work.

Want to see the full leaderboard? Start with the Groundtruth benchmark.