TL;DR
- An agent that reads files, runs searches and calls tools cannot be scored the way a single model response is scored. The same model in two different setups produces two different results, and a single number cannot say which part did the work.
- Useful agent evaluation measures five things: the final answer, tool use, the path taken, cost and latency, and whether the evaluation itself stayed honest. The last one is the one most teams skip.
- This guide covers the metrics, how to pick or build a benchmark, how to keep an LLM judge auditable, where the common frameworks fit, and the numbers we got when we measured eight agents on the same 150 questions.
Key Takeaways
- A score belongs to the model and its harness together. Eight models in one identical harness used their tools so differently that tool mix alone said nothing about who scored well.
- Decide how small a difference your benchmark can actually see, before reading any result. On 50 questions per district, ours resolves 0.27 to 0.45 points. Smaller gaps are noise.
- An LLM judge needs a machine-checkable output contract on top of a good prompt. Ours must return JSON that passes eleven checks, and an answer it never grades validly is left unscored instead of guessed.
- An agent with file tools can read your grading key. In our runs it reached the rubric folder on 13 of 50 questions before we moved the files out of the repository.
- Most public agent benchmarks measure a general capability, not yours. A domain benchmark of 50 questions built from your own documents tells you more about your use case than a saturated leaderboard.
A chat model gets a prompt and returns text. Grading it is comparatively easy, because the thing you are grading is the thing you can see.
An agent is different. It reads files, greps, runs shell commands, calls APIs, sometimes spawns sub-agents, then writes an answer. Two runs of the same agent on the same question can take different paths.
Change the system prompt, the retrieval layer or the tool list, and the answer changes even though the model did not. Everything that decides what the model sees and what it may call is what we call the harness - designing one that tolerates a model’s mistakes is its own discipline.
This guide is about measuring that whole system, using numbers from our own tests, in which eight agents answered 150 questions on Groundtruth, our own dynamic benchmark, designed to test their skills on geological datasets.
Agent evaluation is not model evaluation
The first thing that breaks when you move from models to agents is attribution.
We ran eight models inside one harness, opencode, with the same three tools over the same staged copy of each district’s documents. Same questions, same judge, same everything except the model. The transcripts show how differently they used that identical toolkit.
Gemini 3.1 Pro Preview made 67% of its tool calls in the shell. GPT 5.6 Sol made 0.4%. Grok 4.6 made 19.0 tool calls per question and finished mid-table. Claude Sonnet 5 took second place with 9.9.
Tool mix did not predict the score either. GPT 5.6 Sol and Grok 4.6 used almost the same mix and finished 0.77 points apart.
If a comparison changes both the model and the harness, the gap between the two rows has two causes and no way to separate them. That is why our leaderboard has a second track that pins the model to one reference id, so a submission’s difference from the unassisted baseline can be credited to the harness. The full setup is in Benchmarking an Agent Harness With the Model Held Fixed.
The practical rule: change one layer at a time, and record which layer you changed.
The five things worth measuring
Most agent evaluation writing covers this table’s first two rows. The last is what quietly invalidates results.
| Layer | What it answers | How it is usually measured | Where it breaks |
|---|---|---|---|
| Final answer | Did the agent produce the right outcome | Rubric scoring, exact match, LLM judge | Fluent answers that score well on a loose rubric |
| Tool use | Did it call the right tool with the right arguments | Tool-call accuracy, schema and argument validity | Correct answer reached by a wasteful route, or the reverse |
| Trajectory | Was the path sensible | Step count, loop detection, plan adherence, trajectory match against a reference path | Reference paths go stale, and several good paths exist |
| Cost and latency | What did the answer cost | Tokens, dollars, wall-clock, tool calls per task | Averages hide the long tail that actually hurts in production |
| Evaluation integrity | Did the measurement stay honest | Contamination checks, transcript audits, judge validation | Silent. Nothing in the output looks wrong |
We measure the first, fourth and fifth directly, and the second from saved transcripts. We do not score trajectories against reference paths, because in open-ended research questions there is rarely one correct route.
End-to-end, trajectory and component-level evaluation
Three shapes of evaluation show up repeatedly, and they answer different questions.
End-to-end. Give the agent the task, take the final answer, grade it. This is the only shape that tells you whether the system is useful. It is also the only one that survives a rewrite of the internals, which makes it the stable baseline to keep across versions.
Trajectory. Grade the sequence of steps: which tools, in what order, with which arguments. Useful when the path matters for compliance or cost, and when a task has a canonical route. The cost is maintenance, since every legitimate change to the agent’s strategy invalidates the reference trajectories.
Component-level. Grade one piece in isolation: retrieval quality, a planner’s plan, one tool wrapper’s output. This is where regressions become debuggable, because a component test names the culprit while an end-to-end score only says the number moved.
A workable split: end-to-end scores as the reported number, component tests as fast CI feedback, and trajectory inspection for diagnosis when a score moves without obvious cause. We keep every candidate’s raw event stream, one file per question, for exactly that reason.
Metrics that survive contact with real runs
Score with a gate, not a checklist
Our rubrics are worth 0 to 10 per question, split into components with fixed point values. One component is the gate. If the answer misses the gate concept, the total is 0 no matter how good the rest is.
After the gate passes, each remaining component is judged on its own, positively: nothing is deducted for style or omission, and a component is zeroed only if the answer directly contradicts it.
This matters for agents specifically, since a long answer tends to cover ground: without a gate, a confident survey of everything adjacent to the question collects partial credit on every component and lands at a respectable score while missing the point.
Report the gate pass rate next to the mean
A mean score of 6.8 can be a model that answers everything moderately, or one that nails half the questions and fails the gate on the rest. The gate pass rate tells them apart.
Pointwise for leaderboards, pairwise for decisions
| Pointwise | Pairwise | |
|---|---|---|
| Question asked | How well does this answer meet the rubric | Which of these two answers is better |
| Output | A number from 0 to 10 plus component detail | A verdict from A much better to B much better |
| Depends on another system | No | Yes |
| Can rank a leaderboard | Yes | No absolute value to rank |
| Judge calls per question | One | Two, with the order swapped |
| Best for | Tracking progress over time | Deciding whether a change beat the current setup |
Pairwise comparison has a failure mode worth knowing: judges favour one position regardless of content. Running the same pair twice with the answers swapped, and treating a flipped verdict as no preference, is the cheap defence.
Know your resolution before you read the result
Fifty questions is a small sample. On ours, the smallest mean gap distinguishable from noise was 0.27 points on one district, 0.33 on another and 0.45 on the third. A harness that improves a score by 0.3 points on the third district has not demonstrated anything.
Publish confidence intervals alongside means, and when two systems fall inside the noise floor, call them indistinguishable instead of ranking them.
Cost belongs in the table
Agent runs vary in cost by an order of magnitude for similar quality. In our run, generating 150 answers cost $7.47 with the cheapest model and $54.97 with the most expensive, and the expensive one did not win. We log every paid call, including rejected and retried judge calls, so a run that dies halfway still leaves an accurate bill.
Keeping an LLM judge auditable
If a language model assigns your scores, the judge is part of your measurement instrument and deserves the same scrutiny as the agent.
Give the judge an output contract. Ours must end its response with a JSON block: the gate decision, every component with awarded and maximum points and a reason, the total, and an adjudication flag. A validator checks eleven rules before the score is accepted, including that every component is reported, awards stay within their maximum, and a failed gate zeroes the total.
Re-ask, then leave it unscored. A response that breaks the contract is rejected and re-graded, up to three attempts, after which the question is recorded as unscored rather than filled with a guessed number. Network failures are handled separately, with up to four HTTP attempts per request.
Keep the reasoning. We store the judge’s full response alongside the number, so any score can be re-read later. A score with no attached reasoning cannot be audited.
Let the judge escalate. An answer on an unanticipated route gets an adjudication flag and a note, and goes to a human.
Disclose the family overlap. Our judge is a model from a family that also appears on the leaderboard - a real conflict, disclosed in the methodology rather than left for readers to notice.
The rubric design behind the contract is covered in Dynamic Benchmarking: How We Did It.
Choosing or building a benchmark
Public agent benchmarks
Public benchmarks are useful for tracking the field and for a first sanity check. These include SWE-bench for resolving real GitHub issues, tau-bench for tool use in customer-service style interactions, GAIA for general assistant tasks, and AgentBench for a spread of interactive environments.
Their limits are structural: scores drift upward as tasks leak into training data, a general task set says little about your documents and workflow, and a benchmark everyone optimises against stops discriminating once the leaders bunch at the top.
Domain benchmarks
A benchmark built from your own material answers the question you actually have. Ours is source-grounded: every question comes from a specific passage in a real exploration report, and every rubric is derived from the underlying principle rather than the sentence. A domain expert vets both.
The argument for generating benchmarks per corpus instead of maintaining one fixed set is in Ground Truth: Dynamic Benchmarking in Highly Specific Fields.
Obscure source material matters here, because questions answerable from general pretraining measure recall rather than reasoning over your data. And questions should be written before the answers, ideally by a different agent, so the rubric does not simply describe the answer someone already had in mind.
The contamination problem nobody plans for
Here is the failure that surprised us most. When the candidate is an agent with file and shell tools, and the rubric lives in the same repository as the harness, the grading key is inside the agent’s search space.
With only the authoring skill hidden, a 50-question run reached the rubric folder on 13 questions and had grading text in its context on 8 of them. Renaming the folder in place did not help: the agent read it out of the new name on 8 of 50. Its own path-deny setting restricted nothing in the version tested.
What worked was making the files absent, by moving them outside the repository for the duration of a run and restoring them on every exit path, including on SIGTERM.
Prevention is not enough on its own. Every answer’s transcript is audited for grading-key access before its score counts, and that audit has caught one leaked answer, which was regenerated. The full account is in Your AI Agent Can Read Its Own Answer Key.
If you take one operational habit from this guide, take this one: assume every file the benchmark’s account can read is readable by the candidate, and verify it from transcripts rather than from configuration.
Where the common frameworks fit
Agent evaluation tools cluster into four groups. We run our own harness, so what follows describes what each group is for, not a benchmarked comparison between them.
| Category | What it gives you | Examples |
|---|---|---|
| Eval libraries in your test suite | Metrics and assertions you run like unit tests, in CI | DeepEval, Ragas for retrieval pipelines, OpenAI Evals |
| Research and safety frameworks | A structured task, solver and scorer model, sandboxing, and a log viewer for reproducible runs | Inspect AI, from the UK AI Security Institute, with its Inspect Evals collection covering benchmarks such as GAIA and SWE-bench |
| Tracing and observability platforms | Traces of live traffic, datasets built from production runs, online scoring and dashboards | LangSmith, Arize Phoenix, Braintrust |
| Your own harness | Exactly the metrics your domain needs, with full control of judge, rubric and isolation | Groundtruth, in our case |
Choosing between them is mostly a question of where your evaluation lives. Libraries suit teams who want evaluation to fail a build. Platforms suit teams whose main risk is in production behaviour they cannot see.
Building your own is worth it for domain reasoning no general metric captures. It also gives you control over how answers are isolated from grading material.
Whichever you pick, the parts that determine whether the numbers mean anything are the ones you own: the tasks, the rubric, the judge contract, and the isolation.
A practical checklist
- Write down what a good outcome is before choosing metrics, in the words of someone who does the work.
- Change one layer at a time, and record model, harness, prompt and tool list with every result.
- Score end to end for the headline number, and add component tests for the parts that break.
- Use a gate so an answer that misses the point cannot collect partial credit.
- Report the gate pass rate next to the mean.
- Give the judge an output contract and a validator, and leave unscorable answers unscored.
- Swap the order in pairwise comparisons and treat flips as ties.
- Calculate the smallest difference your sample size can detect, and publish it with the results.
- Keep cost and tool calls per task in the same table as quality.
- Isolate grading material from an agent’s filesystem, then audit transcripts to confirm it held.
- Re-run a stable reference system periodically, so you can tell a model regression from a benchmark drift.
Frequently asked questions
What is AI agent evaluation?
The measurement of a system that plans and acts, rather than of a single model response: the quality of the final outcome, how the agent used its tools, the cost of getting there, and whether the measurement itself was sound.
Which metrics matter most for agents?
Task success against a rubric with a gate, gate pass rate, and cost and tool calls per task. Trajectory metrics help when a task has a canonical route, but cost more to maintain than most teams expect.
How many test questions are enough?
Enough that the difference you care about is larger than your noise floor. Fifty questions per domain resolved 0.27 to 0.45 points for us - fine for systems a point apart, useless for systems a tenth apart.
Can an LLM judge be trusted to score agents?
With constraints: a machine-checkable output contract, validation against the rubric with retries, unscorable answers left unscored, stored reasoning, and disclosure of any family overlap between judge and candidates.
How do public benchmarks compare with a private one?
Public benchmarks track the field and give a first sanity check. A private, source-grounded benchmark built from your own documents shows whether a system works on your material, and does not saturate once everyone optimises against it.
Where to go next
The Groundtruth benchmark, its rubrics and its submissions are public: current results on the Groundtruth leaderboard, project overview on the Groundtruth benchmark page.
Deep dives: Your AI Agent Can Read Its Own Answer Key (contamination), Benchmarking an Agent Harness With the Model Held Fixed (attribution), Dynamic Benchmarking: How We Did It (rubric design), Which AI Model Is Best for Your Geology? (the results), and Survival is the Only Reward: The Observer Effect in AI Benchmarking (agents whose environment keeps changing).
On the building side: How Not to Build a Slop Cannon on a harness that tolerates model error, and Self-Improving AI Agent Architecture on where the evaluator sits in an agent that updates itself.