Which AI Model Is Best for Your Geology?

TL;DR

• Standard academic AI benchmarks are stale and largely irrelevant to a real decision like spending $100,000 on an AI system for exploration, because geology is hyper-local: a general fact may not hold in a specific terrain.
• The Groundtruth Dynamic Benchmark generates fresh questions and scoring rubrics from your own documents instead of testing against a fixed, generic question bank.
• Three free sample test sets (Technical, WAMEX and USGS) sit alongside support for fully custom datasets. Questions test finding facts, reconstructing geological events, reconciling sources that disagree, and recognising when the evidence runs out.
• In our bake-off, Kimi K3 scored highest, but the top models’ confidence intervals overlap, so the benchmark sorts models into tiers rather than a strict ranking. Once the top tier is tied, cost decides.

Key Takeaways

• Benchmark relevance, not benchmark existence, is the real problem in specialist domains. A high score on a generic geology benchmark says almost nothing about whether a model will work on a specific company’s own archive, which is why the tool generates a new benchmark per dataset rather than reusing one.
• Once accuracy differences between top models become statistically indistinguishable, cost becomes the deciding factor. That is why the price-performance frontier, not the leaderboard, is the more actionable output of the exercise.
• The benchmark’s real long-term value is as a standing tool for re-evaluating vendor claims over time, rather than a one-time model selection exercise. Rerunning the same frozen benchmark against a proposed cheaper model is only possible because the benchmark is generated from the buyer’s own data rather than a shared public one.

If you gave an AI the same messy exploration archive you would give a consulting geologist, how much would you trust what came back?

That is the real version of the question “which AI model is best for your geology?”, and a public leaderboard can’t answer it.

Why Standard AI Benchmarks Don’t Work for Geology

Traditional AI benchmarks are basically an exam paper everyone has had access to for years. Eventually you don’t really know whether a model has learned the subject or learned the exam. The papers themselves are not always right either: a 2024 re-check of MMLU, one of the most widely used benchmarks, estimated that 6.49% of its questions contain errors, rising to 57% in its virology subset. This isn’t the only problem when it comes to working out which model is best for geological work, however.

Suppose you’re an exploration manager with 15 years of reports, logs, maps, assays and consultant studies sitting in SharePoint. Someone wants you to spend $100,000 integrating an AI system. The vendor tells you that Model X scored 83% on some academic geology benchmark. You say: who cares?

Because every terrain is so unique, it can be actively damaging to rely too much on general knowledge. Sure, bismuth is often a gold pathfinder, but if you’re an explorer you want to know whether it’s a pathfinder in your specific geology. We set out the full case for benchmarks built for highly specific fields separately.

That’s why we made the Groundtruth Dynamic Benchmark. Rather than asking a standard set of questions to test the sector knowledge of a model, Groundtruth generates a fresh set of questions and scoring rubrics from your own dataset. That lets you compare models and agent harnesses on the data that matters to you.

How Does the Groundtruth Benchmark Work?

If you want to test the skill and versatility of a new model or agent harness, the Groundtruth benchmark provides three diverse test sets free: Technical, WAMEX and USGS. If you want to compare model or agent skills on your own data, the system will generate a custom question set and scoring rubric based purely on that dataset. The generation process is set up such that some questions will require finding facts. Others will require reconstructing sequences of geological events, distinguishing competing interpretations or deciding what the evidence actually supports.

Can it find the evidence?

“Which formations make up this succession?”

Can it reconstruct what happened?

“What happened first: metamorphism, deformation or gold mineralisation?”

Can it reconcile geologists who disagree?

“One report says there is a major shear. Another says there isn’t. What does the evidence actually establish?”

Does it know when the evidence runs out?

“Two reports use different names for apparently similar greenstone belts. Can the model resist simply declaring them identical?”

Running the test

Once you have your data and your question set, you’re ready to test. Pick a model and add in a custom agent harness if you want one, such as our own Geocluster Research Harness, then let it run. Once complete, the grading AI will check the answers, making sure that they match the grading rubric rather than simply being persuasive-sounding verbiage.

Which AI Model Won the Geology Benchmark?

Kimi K3.

But that’s not the full story.

While Kimi got the highest one-off score on the Groundtruth leaderboard, statistically there was enough overlap between the three top models that they can more or less be treated as equally good. This being so, the question of price is an important one.

AI model scores and cost per question

Costs include both generation and judging. Cost per question is the total divided by the 150 questions across the three districts.

If we compare price and accuracy, we end up with four models offering roughly equivalent ROI: Deepseek V4 Pro, GPT 5.6 Sol, Claude Sonnet 5 and Kimi K3… at least until Ox Alpha was released, which shifted the Pareto frontier drastically.

What This Means for Choosing an AI Model

This means that the exploration manager from the opening paragraphs can now run:

• Commercial models + vanilla retrieval
• Commercial models + their proposed RAG system
• Commercial models + an agentic search harness
• Whatever the vendor is proposing

against a custom benchmark derived from their own documents.

How to Choose the Right AI Model for Your Exploration Data

The answer to “which AI model is best for your geology?” is not a name on a leaderboard. It is the result of three steps.

Start With Your Own Documents, Not a Leaderboard

Generate a benchmark from your own reports, logs and scanned geological maps rather than relying on a public score. A model that ranks first on a generic test may still misread your stratigraphy, your naming conventions or your consultants’ disagreements. Only a question set built from your archive tests what you actually need.

Weigh Price Against Performance, Not Accuracy Alone

Once the top models are statistically tied, the Pareto frontier of cost against score decides. In our run, Ox Alpha landed in the top tier at a fraction of the others’ cost, which moved the frontier on its own. A model that is a few tenths of a point ahead but costs several times more is rarely the better choice.

Re-run the Benchmark Whenever Something Changes

A new model release, a vendor proposing a cheaper migration, or a new dataset are all reasons to test again. Six months later, when someone says “we should migrate to Model Y because it’s 40% cheaper,” you can rerun the same frozen company benchmark and find out whether that change actually degrades performance on your work.

Conclusion: Which AI Model Is Best for Your Geology?

On our three sample districts, a top tier of four models performed about equally well, and cost separated them. On your own archive the answer may differ, which is exactly why the benchmark is built from your documents.

The per-district results for all seven models are on the Groundtruth leaderboard. The benchmark itself is open source (MIT), so you can run the same test on your own reports before committing to any model.

FAQs

What is a dynamic AI benchmark?

A dynamic AI benchmark generates its questions and scoring rubrics from a specific set of documents, instead of reusing a fixed public question bank. Groundtruth reads your corpus, writes questions grounded in it, and checks every answer against the source. Because the questions are new, a model can’t have memorised them during training.

How is Groundtruth different from public benchmarks like MMLU?

MMLU is a fixed set of questions that models may have seen in training, and it tests general knowledge. Groundtruth builds a fresh question set from your own documents, so it tests whether a model can reason over the material you actually work with. It also grades against a rubric with a hard gate, so a fluent but wrong answer scores zero.

How often should you re-run an AI model benchmark?

Whenever something that could change the answer changes: a new model release, a vendor proposing a cheaper model, a change to your retrieval or agent setup, or new data in your archive. Because the benchmark is frozen and built from your documents, a rerun gives a like-for-like comparison with your earlier results.

What’s the difference between accuracy and cost-adjusted model value?

Accuracy is how well a model scores. Cost-adjusted value weighs that score against what it costs to get it. When several models’ scores are statistically tied, the cheaper one gives more value for the same result. In our run, cost per question ranged from about $0.04 to $0.37 among models of similar accuracy.

Can I test my own agent harness or RAG system with Groundtruth?

Yes. You can run any model with your own retrieval or agent harness against the same question set, and compare it with the plain model. The leaderboard also has a harness track, which pins the model so that any difference in score comes from the harness itself rather than from the model underneath.

Do I need my own dataset to use Groundtruth, or can I use the free sample sets?

You can start with the three free sample sets, Technical, WAMEX and USGS, to compare models and harnesses straight away. To learn which model works best on your own ground, generate a custom benchmark from your own documents. The same pipeline builds both.