TL;DR
- A leaderboard score belongs to a model and a harness together. Comparing harnesses that run on different models mixes the two effects, and there is no way to separate them afterwards.
- On Groundtruth, our benchmark that tests AI agents by having them answer real mineral-exploration questions, eight models ran in the same opencode harness with identical tools, and their tool use still differed sharply. Gemini 3.1 Pro Preview made 67% of its tool calls in the shell; GPT 5.6 Sol made under 1%.
- The harness track pins the model to one reference id, currently GLM 4.7 via OpenRouter, so a submission’s difference from the direct-call baseline can be attributed to the harness.
Key Takeaways
- A single leaderboard number can’t separate a model’s effect from its harness’s effect. If one row is Model A in Harness X and another is Model B in Harness Y, the gap between them has two causes with no way to untangle them afterwards.
- Holding the model fixed and identical tools does not make agents search alike. Across eight models in the same opencode harness, tool-call mix and effort varied sharply, and mix didn’t even predict score - GPT 5.6 Sol and Grok 4.6 used almost the same mix and still finished 0.77 points apart.
- The harness track isolates the harness by pinning everything else: the reference model (currently GLM 4.7), the three districts and rubrics, the judge model and temperature, and the result format - so a harness submission is compared against the same model’s own unassisted baseline.
- Fifty questions per district limits how small a difference the benchmark can confirm (0.27-0.45 points depending on district), so a harness result should be read against that noise floor, not treated as exact.
A team choosing an AI setup for exploration work is choosing two things at once. One is the model. The other is everything wrapped around it: the retrieval layer, the tools the model may call, the system prompt, and the agent loop that decides when to search again and when to answer.
We call that wrapper the harness - designing one that tolerates a model’s mistakes is its own problem, separate from picking the model.
A single number on the Groundtruth leaderboard - our public ranking of AI agents on the same set of mineral-exploration questions - describes the pair, though. If one row is Model A in Harness X and another is Model B in Harness Y, the gap between them has two causes. That leaderboard handles this with two tracks, and this post explains why the second one exists.
One harness, eight different search strategies
The model track already holds the harness constant. Every model we ran ourselves used the same agentic harness, opencode, with the same tools (file reading, search and a shell) over the same staged copy of each district’s documents: 150 questions, 50 per district.
The saved transcripts show how differently the models used that identical toolkit. Share of each model’s tool calls:
| Model | read | grep | bash | glob | Mean score |
|---|---|---|---|---|---|
| Kimi K3 | 40% | 27% | 31% | 2% | 8.81 |
| Claude Sonnet 5 | 42% | 2% | 55% | 1% | 8.62 |
| GPT 5.6 Sol | 57% | 32% | 0.4% | 10% | 8.50 |
| OxAlpha | 39% | 29% | 27% | 5% | 8.44 |
| Deepseek V4 Pro | 56% | 24% | 14% | 2% | 7.79 |
| Grok 4.6 | 57% | 32% | 0.2% | 12% | 7.73 |
| Gemini 3.1 Pro Preview | 11% | 21% | 67% | 0.5% | 7.55 |
| GLM 4.7 | 46% | 29% | 17% | 8% | 6.81 |
The remaining share is sub-agent delegation and web fetches, under 4% for every model.
Two things stand out. First, tool mix does not explain score. GPT 5.6 Sol and Grok 4.6 have almost the same mix and finished 0.77 points apart. Kimi K3 and OxAlpha look alike too, and sit 0.37 apart.
Second, effort varied as much as style. Grok 4.6 made the most tool calls, 19.0 per question, and landed mid-table. Claude Sonnet 5 took second place with 9.9 calls per question, the fewest of the leading three. Gemini 3.1 Pro Preview made 9.6 calls per question and had the highest total spend of the run, $54.97, nearly all of it generation.
Inside one harness, each model brings its own way of searching. Part of what you would credit to a harness in a mixed comparison is really the model’s own behaviour. That is the case for holding the model fixed when the harness is the thing under test.
Two tracks
Model track. Any model, run as submitted. The ranking describes the model in the setup its submitter used. For the rows we ran, that setup is plain opencode with no custom scaffolding.
Harness track. The model is pinned. The benchmark dataset carries a harness_track.json file whose reference_model_id is currently openrouter/z-ai/glm-4.7. A submission counts toward the harness track only if the candidate_model_id in its meta.json matches that id exactly and it names its harness.
Everything else about the setup is the submitter’s choice: retrieval, tools, prompts, sub-agents and the loop. Meet the Geocluster Research Harness covers how ours is built.
The comparison point exists already. The model track contains GLM 4.7 run with no custom harness on all three districts, scoring 6.76 on Technical, 6.98 on WAMEX and 6.70 on USGS, 6.81 overall. In the leaderboard’s harness view those rows appear as the “no harness” baseline, so any harness submission is compared with the raw model on the same questions under the same judge.
GLM 4.7 sits at the bottom of the model table, which leaves room to improve on it. It is also inexpensive to run: generation for all 150 questions cost $7.47 in our run.
The reference model is stored as a configuration value so it can be revisited, for example if a cheaper or more widely available model becomes a better default. Changing it does not invalidate earlier harness submissions, but they stop counting toward the current harness board until someone re-runs them against the new reference.
What stays fixed
The harness track isolates the harness by holding the rest of the measurement constant:
- the three 50-question districts and their rubrics, published as a Hugging Face dataset;
- the judge,
openai/gpt-5.5, scoring every answer pointwise from 0 to 10 at temperature 0; - the reference model id;
- the result format, which is the unedited per-question
scores.jsonproduced by scoring.
One setting is easy to miss. When a candidate runs through opencode, its sampling parameters come from the local opencode configuration and not from the benchmark code, and some reasoning models accept no temperature setting at all. Record the sampling settings in the submission’s notes field so someone else can reproduce the run.
Entering the harness track
- Run your harness with the reference model over one district’s questions and save one answer per question.
- Score the answers pointwise with the benchmark’s judge and that district’s rubric. Start with WAMEX or USGS - Technical’s rubric is on a newer format that currently crashes the scorer, tracked as a known issue.
- Copy the submissions template into
submissions/harness-benchmark/<slug>/in the submissions dataset, add the uneditedscores.json, and fill inmeta.json, includingcandidate_model_id,harness_nameandharness_repo. - Open a pull request.
Results are self-reported and we do not re-run them. The leaderboard flags any submission whose question ids do not match the rubric named in its meta.json. Every row links to its raw scores and to the harness repository, so a reader can check the work.
Reading a harness result
Fifty questions per district limit how small a difference the benchmark can confirm. The smallest mean gap distinguishable from noise, measured on a representative model pair, was 0.27 points on WAMEX, 0.33 on USGS and 0.45 on Technical.
A harness that lifts GLM 4.7 by 0.3 points on Technical has not demonstrated an improvement on that district, while the same lift on WAMEX has. It is worth reading all three districts, because the lower half of the model table changes order from one corpus to the next.
The harness view is on the Groundtruth leaderboard under “By harness”, with every submission linking back to its harness repository for anyone who wants to check the work.