Benchmarking on Eigenform Articles

Which AI Model Is Best for Your Geology?

Mon, 24 Aug 2026 00:00:00 +0800

TL;DR

Argues that standard academic AI benchmarks are stale (“an exam paper everyone has had access to for years”) and largely irrelevant to a real decision like spending $100,000 integrating an AI system into an exploration workflow, since geology is hyper-local and a general fact, such as bismuth being a gold pathfinder, may not hold in a specific terrain.
Introduces the Groundtruth Dynamic Benchmark, which generates fresh questions and scoring rubrics directly from a user’s own document set rather than testing against a fixed, generic question bank.
Offers three free sample test sets (Murchison WAMEX gold reports, USGS Southwestern US reports, Canadian technical reports) alongside support for fully custom datasets, with generated questions spanning four types - finding facts, reconstructing sequences of geological events, reconciling sources that disagree, and recognising when the evidence doesn’t support a firm answer - graded against a rubric rather than accepted for sounding persuasive.
Reports the results of a six-model bake-off: Kimi K3 scored highest overall, but confidence intervals overlap enough among the top three models that the benchmark can only reliably sort models into tiers rather than a strict ranking, and that tiering held consistently across different source-document types.
Given the near-tied top performance, shifts the comparison to cost: generation cost dwarfs judging cost and varies sixfold between models doing the same task, and plotting price against performance puts four models on a shared frontier of roughly equivalent return on cost - until a newly released model, Ox Alpha, is added, whose free-tier pricing puts it alone on a sharply shifted frontier ahead of everything else.
Concludes with the practical payoff: an exploration manager can benchmark any model-plus-retrieval or model-plus-agent combination against their own frozen, company-specific test set, and rerun that same benchmark later to check whether a proposed cheaper migration actually degrades performance on their own work, rather than relying on vendor claims or generic scores.

Key Takeaways

Groundtruth Dynamic Benchmarking (Geology Edition) v1.0.0 Is Live

Fri, 31 Jul 2026 02:00:00 +0800

TL;DR

Announces v1.0.0 of the Groundtruth Dynamic Benchmarking System on GitHub: not a fixed set of benchmark problems, but a pipeline for generating reliable benchmark questions and scoring rubrics for whatever corpus you’re working with.
Ships a working example grounded in a real, openly licensed corpus - 34 mineral deposit records from the Yudnamutana Copper district, South Australia - runnable end-to-end with a single script after cloning the repo and adding API keys.
Uses a rubric format built to resist grade inflation: a hard gate per question, non-overlapping graduated components, required concepts, and do-not-credit lists, with the underlying mechanics covered in the companion post “Dynamic Benchmarking: How We Did It.”
Grading is structured and auditable: the judge returns a full breakdown of which components it credited and why, and flags anything it can’t verify from the rubric alone rather than guessing.
The corpus-to-rubric authoring pipeline itself ships in the repo, so anyone can point it at their own corpus and build their own benchmark edition.

Key Takeaways

Ground Truth: Dynamic Benchmarking in Highly Specific Fields

Fri, 31 Jul 2026 01:00:00 +0800

TL;DR

Standard benchmarks like IMO results or HumanEval work for general skills, but don’t tell you whether a model fine-tuned on a narrow, specific corpus - such as geological reports on one region of Western Australia - actually learned anything from that corpus.
The piece distinguishes capability benchmarks (“what can this model do”) from knowledge-acquisition benchmarks (“what did this model learn after reading this corpus”), arguing continual fine-tuning projects need the latter, which existing benchmarks conflate away.
Its proposed fix, “dynamic benchmarking,” replaces hand-authored benchmarks with a benchmark generator: a candidate model is examined by a panel of frontier models with corpus and web access that identify what to test, write questions and rubrics, and only then grade responses.
This is explicitly distinguished from LLM-as-judge: the examining panel doesn’t need to be smarter than the candidate, only better resourced and structured to reach a consensus standard.
Using the Cue Victory gold field as a case study, the piece separates three confounded things a naive benchmark would blur together - innate pretrained knowledge of a region, general data-analysis skill, and corpus-specific learning - and argues only the third is what a continual-learning benchmark should isolate.
The pipeline itself runs: read the corpus, build an internal understanding, identify testable concepts, generate questions and reference answers and rubrics, verify every answer against source evidence, then hold out the benchmark, with pairwise comparison against the untuned foundation model as the preferred scoring method.

Key Takeaways

Dynamic Benchmarking: How We Did It

Fri, 31 Jul 2026 00:00:00 +0800

TL;DR

The post documents building a benchmark to check whether a LoRA adapter fine-tuned on regional geology (the Cue Victory goldfield, renamed during testing to avoid pre-trained knowledge leakage) actually improved geological reasoning, rather than just improving recall of its own training data.
Questions were generated from historical tenement reports by a large model, each traceable to a specific source passage, with grading criteria derived top-down from the underlying geological principle rather than read off the source text, and every question and rubric vetted by a human geology expert before use.
The fine-tune was scored two ways: pointwise against a rubric (hard gate, then graduated non-overlapping components, then a do-not-credit list for plausible wrong answers), and pairwise head-to-head against the untrained base model, with each comparison run twice with the order swapped to cancel position bias.
The corpus-to-question-to-rubric process was codified into a reusable staged pipeline: map the corpus first, establish a source-authority contract for conflicting material, write questions before answers (ideally with different agents), and calibrate every rubric block against a gate-fail, a bare pass, and a strong response before trusting it.
Five practical problems are laid out in the order they were hit: LLM judges reward fluency over correctness unless the rubric closes that gap; source material must be obscure enough that it isn’t just testing the base model’s pretraining; pointwise and pairwise scores can disagree, so both need to be run; failed generations need capped, logged retries rather than being swept away; and a single benchmark run is noisy enough that results need averaging across repeated runs.

Key Takeaways

Survival is the Only Reward: The Observer Effect in AI Benchmarking

Tue, 21 Jul 2026 10:00:00 +0700

TL;DR

Frames the core problem as benchmarking an agent whose reward is an emergent property of a constantly changing open-world environment rather than a fixed scalar - a setting where reward hacking is the intended behaviour, not a bug to eliminate - which breaks the standard ML-evaluation assumptions of a fixed task, a fixed reward, and “more reward means improving.”
Even the reward itself turned out to be hard to observe: measuring a container’s actual disk usage from outside proved imprecise past the kilobyte level, and an in-container measurement tool got deleted by the agent itself, since deleting unrecognised files is its whole job - the published results are explicitly built on imprecise measurements, with a genuine fix (giving agents a sense of what memory is “theirs”) still unsolved.
A major driver of performance - raw efficiency gains that cut data-gathering time from nearly a week to under ten hours - wasn’t captured by the original composite metric at all, and only became visible after adding an inverse-hours term and z-scoring generation-over-generation change; the stated lesson is to instrument far more than seems necessary from day one, since a generation that has already run can’t be retimed retroactively.
Performance was initially reported as a single number per generation, until an outside reviewer pushed for variance and confidence bounds, leading to a rerun of each saved checkpoint 100 times - a method the piece calls a kludge given the resulting limited statistical power - and this surfaces a deeper mismatch: standard i.i.d. sampling assumptions don’t really apply when every generation and environment is procedurally distinct, so the team argues the variance reflects genuine ecological diversity rather than pure noise, and proposes measuring it across swarms of agents rather than repeated runs of one checkpoint in future work.
The pass@1 metric, added almost as an afterthought, turned out to be the most informative one for an unexpected reason: it fell steadily even as real task success rose, revealing that agents had evolved to deliberately write failing code first to harvest informative error messages - a behaviour the piece stresses emerged through selection, since lineages that happened to do this produced more useful training data, rather than through any reasoned decision by the agents.
Closes by arguing that evaluation paradigms built around fixed-skill, fixed-time competence will need to give way to methods for measuring adaptability and strategic tradeoffs as test-time adaptation becomes more common, framing physical, hard-to-fake consequences as a better teacher and a better benchmarking subject than either a single number or one model judging another’s output.

Key Takeaways